Testing Introduction and Basic Process

Software testing is a discipline that ensures the quality and reliability of software products before they reach users. This article introduces why testing is necessary, what it is, the core principles that govern testing work, and the practical tools and processes used in the industry.

Why Testing Is Needed

In many cases:

• Whenever a mobile game like Honor of Kings releases a new hero or an existing hero becomes too powerful — dominating every match — players complain that the hero is broken and demand a nerf.
• When playing popular battle royale games, if the system fails to protect players from being eliminated before they even land, or getting killed without ever seeing an opponent, players rightfully shout “this is a bug!”
• Or consider mobile payment: if scanning a QR code causes your app to show a 998 yuan deduction while the merchant receives nothing — that is a serious bug.

A piece of software goes through many stages before it is born, with different people involved at each stage. The final product will inevitably have problems. To ensure software usability, we must go through a testing phase to reduce issues.

No programmer dares claim their code has zero bugs — yet the software we use every day rarely shows obvious bugs. That is inseparable from software testing.

Therefore, a software product providing business access must pass rigorous testing through multiple test environments before going live. Just like games that go through closed beta before public release, companies verify program correctness at each stage.

Why Developers Don’t Test Their Own Code

Developers are not incapable of testing — in fact, some testers previously worked as developers.

The point is that software testing and software development are two different technical directions within the software industry. A mediocre developer is not as good as a professional tester when it comes to finding bugs. This is a matter of specialization.

From a logical perspective, developers spend most of their time thinking about how to implement specific features. Testers, on the other hand, spend most of their time thinking from the user’s perspective about how to expose problems in the software.

From a testing-rigor perspective, a software product is like a developer’s own child. “How could my child possibly have problems? Your child is the one with problems!” This mindset means developers testing their own code may not be thorough enough.

What Is Testing

Glenford J. Myers defined it in his book The Art of Software Testing: Testing is the process of executing a program with the intent of finding errors.

Software experts Weinberg and Cem Kaner also offered their understanding. In Weinberg’s Perfect Software, he states: Testing is a process of gathering information to reduce decision risk. Professor Cem Kaner proposed: Software testing is a technical investigation conducted to provide stakeholders with empirical information about the quality of a product (software, system, or service).

IEEE (Institute of Electrical and Electronics Engineers) and ISO (International Organization for Standardization) also published their views. In 1983, IEEE defined software testing as: The process of using manual or automated means to run or test a system, to check whether it meets specified requirements or to clarify the difference between expected and actual results. This shows that software testing is not only about finding errors but also about verifying that software meets specified requirements. ISO 29119 also attempted to standardize software testing, stating: Software testing should focus on providing information about a software product and finding as many defects as possible, as early as possible in the development process, under given constraints of costs and schedule. Two key points here: finding bugs as early as possible (smaller impact, lower fix cost), and testing under cost constraints (limited resources require prioritization).

To put it plainly: the more code you write, the more you appreciate testing. There is an apt analogy: developers are like building a bridge without guardrails — they know where to walk safely even in the dark. Testers are like adding guardrails to the bridge, letting ordinary users navigate it as easily as the developers who built it. From this perspective, testing is even more important than development.

Summary — the definition of software testing:

Using manual or related tools to perform testing operations on the subject under test, in order to verify whether actual results differ from expected results.

Why Software Testing Emerged

• Software Complexity

  The complexity of program code and the concurrency and complexity of software products continue to increase, raising the bar for correctness verification.

• Industry Competition

  As user expectations and demands rise, news apps alone have Baidu News, NetEase News, Qutoutiao, Toutiao, and many others. Companies all want to be the best, surpassing competitors in usability, aesthetics, engagement, speed, and more. Large companies therefore staff dedicated functional testing, performance testing, and even more powerful test development positions.

The Role of Software Testing

Testing work can discover and fix defects in software, thereby increasing user confidence in the product.

Testing can record data generated during software operation, providing data support for decision-making.

Testing can reduce the risk of problems encountered when developing similar products.

The Development of Software Testing

Domestically, the field is in a period of early growth and rapid development. Large companies pay great attention to testing; early-stage small companies pay less attention. Manual testing remains dominant, supplemented by automated testing.

Internationally, software testing is fairly mature. Software companies value their testing departments greatly. Testing processes and systems are rigorous. Leading companies establish software testing centers to serve the software development of their subsidiaries.

Goals of Software Testing

Expose hidden errors and problems in software through testing, to inform decisions about whether to use the product. Just as when buying a phone, you repeatedly check: How is the CPU performance? How much memory does it have? How is the camera?

From the software developer’s perspective:

Prove through software testing that the software contains no errors or problems,
and build sufficient confidence in the quality of the product.
A successful test is one that relentlessly uncovers software errors and continuously improves the product.
Meeting user requirements is the key to product success.
Ensure delivered products meet user needs.
Find and fix as many bugs as possible before the product goes live.

From the user’s perspective:

Finally, after shaking my phone for so long, someone added me on WeChat!

Common Misconceptions About Software Testing

• Debugging and testing are the same thing.
• The testing team should be solely responsible for ensuring quality.
• Over-reliance on beta testing (acceptance testing).
• Treating testing as a transitional position for new hires.
• Assigning unqualified developers to do testing.
• Focusing on test execution while ignoring test design.
• Believing test automation can solve everything.
• Believing testing can be exhaustive.
• Thinking testing is about proving software correctness.
• Thinking testing is tedious, uncreative work.
• Relying excessively on test metrics to measure software quality.

Main Work of Software Testing

• Review code and evaluate development documents.
• Perform test design; write test documentation, test plans, test schemes, and test cases.
• Execute tests, discover software defects, submit defect reports, and track the defect-fixing process.

Software Testing Principles

Testing principles are rules that must be followed when carrying out testing work:

• Testing proves that defects exist in software. Whatever testing operations are performed, they can only confirm that the current software has defects — not that it is defect-free.
• Exhaustive testing cannot be performed. Some functionality has no way to enumerate all possible cases, so any testing activity has an end time.
• Defects cluster together. First understand the 80/20 rule: for software functionality, core features account for 20%, and non-core features account for 80% (not absolute). In testing, we focus on the 20% core functionality. Therefore, the probability of finding defects in this part will far exceed the 80% non-core portion. That is why most defects we encounter are concentrated in the 20% core functionality.
• Some testing requires special environments. Certain tests depend on extreme conditions that can be difficult to reproduce.
• Testing should be introduced as early as possible. The earlier defects are discovered and resolved, the smaller the impact and the lower the fix cost.
• Pesticide paradox. The same test cases cannot be executed repeatedly, because the software will develop “immunity” to them. For example, if you use 3 * 3 to discover that the code does not equal 9, report the defect, and the developer fixes it, don’t use 3 * 3 again next time — the developer specifically made 3 * 3 = 9, so the same test case will no longer find the bug.
• The absence-of-errors fallacy. No software can be perfect.

Subjects of Software Testing

For the current testing industry, the most commonly tested subject is software (core functionality), but what we need to test is not limited to functional requirement testing. We can divide software into three components:

• Functional collection
• User manual
• Configuration data

A piece of software goes through different processes during its creation. We divide the entire process into different phases, and each phase has corresponding test subjects. What can we test at each phase?

• Requirements analysis phase: various requirements specification documents. For example, can the current technology achieve this? Does similar software exist on the market? Corresponding specification documents are produced.
• Software architecture design: the CTO or lead architect designs the overall software architecture, then generates API documentation (interface testing), which is handed to dedicated developers for specific implementation.
• Coding phase: at this point, the test subject is source code (though this requires high testing skill), so corresponding white-box testing and unit testing are performed.
• System functional use: the main functional testing of the software — the most common type of testing in the industry — where testers act as users to use and test the software.

Software Architecture

Software architecture can be simply understood as a set of ideas used to guide software development. Currently, the two most common architecture patterns are:

• B/S (Browser/Server architecture): browser and server.
• C/S (Client/Server architecture): client and server.

Comparison of the two architectures:

• Efficiency: In B/S architecture, data is processed by the server while the browser only displays results, putting relatively higher pressure on the server. In C/S architecture, the client can handle some data processing, so execution efficiency is higher.
• Security: B/S architecture data travels over HTTP, so its security is relatively lower than C/S architecture.
• Upgrades: B/S architecture only requires upgrading the server side, while C/S architecture requires upgrades on both sides.
• Development cost: Compared to B/S, C/S architecture also requires developing the client side, so costs are higher.

Two additional knowledge points:

Browsers

A browser is essentially a software application installed on an operating system, providing web browsing services to users. The world’s five major browser vendors:

• IE (Windows Internet Explorer): IE4 and above use the Trident engine. Due to its monopoly position, IE went a long time without updates, causing two consequences: IE fell out of sync with W3C standards, and Trident engine bugs accumulated without being fixed. This gave other browsers their opportunity (e.g., Firefox). These issues caused great difficulty for web front-end developers, especially IE6 during a transitional period. Microsoft’s latest browser, Edge, now uses Google’s engine.
• Chrome (Google Chrome): Google Chrome previously used Apple’s WebKit engine, but has since forked off and created its own Blink engine, which is also jointly adopted and developed by Opera.
• Safari (Apple): Uses Apple’s own WebKit engine.
• Opera: Originally used Presto; now uses Blink alongside Chrome.
• Firefox (Mozilla Firefox): Uses Gecko.

Domestic Chinese browsers and their engines:

• Sogou Browser: Compatibility mode (IE: Trident) and Speed mode (WebKit)
• Maxthon Browser: Compatibility mode (IE: Trident) and Speed mode (WebKit)
• QQ Browser: Normal mode (IE: Trident) and Express mode (WebKit)
• 360 Speed Browser: Based on Google (Chromium) and IE engine
• 360 Safe Browser: IE engine

For browsers, the most core technology is the browser engine — worth knowing at a general level.

Images

Common image types:

• jpg (jpeg): Can highly retain image color information; commonly used on internet clients.
• png: This type of image supports transparency.
• gif: Small file size; supports animation.
• psd: Layered image format.

Project Team Structure

A project team is generally led by a project manager who is responsible for making project plans and assigning tasks.

Participants include:

• Analysts
• Designers
• Developers
• Testers
• Configuration management personnel — repository managers for the software development process, including products, documentation, etc.
• SQA (Software Quality Assurance) — monitors the entire software development process.

Software Test Cases

Test cases appear everywhere in daily life. When you buy a phone or a monitor, you test it: power on/off, check for screen backlight bleed, check if buttons work — these are all test cases.

We need to know what to test and how to test it.

What Is a Test Case

Definition:
    A test case (Test Case) is a set of test inputs, execution conditions, and expected results designed
    for a specific purpose, to check whether a specific requirement is met. Large numbers of test cases
    are used to verify the operational effects of software. Test cases are an important basis for guiding
    testing work.

For example, when we buy a computer, we need to test it.
Preconditions for testing:
    Pressing the power button is equivalent to providing a set of test data. The execution condition
    is whether the preconditions for powering on are met — for example, whether the battery has charge,
    or whether an external power source is connected.

Test process:
    Press the power button.

Expected result:
    When we press the power button, if the computer starts up successfully, then — given that power is
    available — successful startup is our expected result.

Why Test Cases Are Needed

Advantages of test cases:
    - Avoid blind testing, improve testing efficiency, and make testing activities orderly and standardized.
    - Reduce the workload of test design and reduce the complexity of regression testing.
    - Based on the number and execution difficulty of test cases, estimate the testing workload and make
      it easier to track project progress and resource allocation.

The Significance of Test Cases

• Test cases are the core of software testing.
  • The importance of software testing is beyond doubt. Test cases guide testing work and are the fundamental guarantee of stable software testing quality.
  • Many factors affect software testing, such as the complexity of the software itself, development quality, and the application of testing methods and techniques. Objective factors also exist, such as personnel changes, environment, and morale.
• Benchmark for evaluating test results. The pass rate and failure rate of test cases are important criteria for determining whether testing is complete and whether the software can go live.
• Ensure no test function points are missed during testing. They provide guidance and direction for testing work.
• Writing test cases helps testers become familiar with requirements and gain a comprehensive and in-depth understanding of system architecture or business processes.

Life Cycle of Test Cases

The Eight Key Elements of a Test Case

• Case ID: Product name - testing phase
• Test item: Corresponds to a functional module
• Test title: Derived directly from refining the test point
• Priority level: High / Medium / Low
• Preconditions: Prerequisites that must be met; otherwise the case cannot be executed
• Test input: Input information to be processed; designed according to specific circumstances
• Operation steps: Clear description of each step; the executor follows these steps
• Expected result: Compare expected output against actual result to determine whether the test subject meets requirements
• Actual result: Fill in the report based on actual results (optional)

Outputting Test Cases

• Excel
• Word
• HTML

Test Granularity

Test Case Granularity

Test cases can be written simply or in great complexity.

• The simplest test cases are outlines of testing, merely pointing out what needs to be tested. If test cases are written too simply, they may lose their purpose. Overly simple test case design does not actually involve “design” — it only records the functional modules to be tested. Their role is merely to serve as a simple test plan during testing, reminding testers of the main functionality to cover.
• The most complex test cases specify every piece of input data, the expected results and verification methods, specific UI operation steps, and the testing methods and tools to use.

Writing test cases that are too complex or detailed brings two problems: efficiency issues and maintenance cost issues. Additionally, overly detailed test cases leave little room for the test executor to think, which can constrain the tester’s thinking. Note: Most testing teams write test cases with a granularity between the two extremes.

The Essence of Test Cases

The essence of test case design (what to test? how to test?) should be to understand and verify requirements during the design process, and to record testing approaches for the software system to guide future testing.

Requirements-based test case design:

• Designing test cases based on requirements-based use case scenarios is the most direct and effective method, because it directly covers requirements, and requirements are the foundation of software. Validating coverage of requirements is the fundamental purpose of software testing.
• Treat test cases as living documents, because requirements are living and changeable. Therefore, the agile principle of “responding to change over following a plan” should be reflected in test case design.

Do not think of test case design as a one-time phase. Test case design also needs iteration — at different stages of software development, return to review and improve test cases.

Test Environment Design

Test Environment (TE)

• A collection of hardware, software, and network environments necessary to run the software under test and complete testing work.
• A stable and controllable test environment allows testers to spend less time completing test case execution.

Test Environment Includes

• Hardware environment
• Software environment
• Network environment
• Test data
• Testing tools

Test Environment Design Principles

• Use real environments as much as possible; minimize the use of simulators and virtual machines.
• Machine configuration must meet the minimum software running requirements.
• Choose mainstream operating systems and software platforms, such as iOS and Android.
• The test environment must be clean, independent, and free from interference — such as unnecessary antivirus software, unnecessary media players, etc. During performance testing, a vulnerability repair program suddenly popping up will definitely affect test results.

Knowledge Required for Test Environments

• Installation and use of common operating systems
• Installing drivers required by programs; resolving driver error issues
• Database usage
• Browser debugging
• Use of simulators and virtual machines
• Use of system image, backup, and restore tools

Software Test Plan

What Is a Test Plan?

A test plan is a document that describes the scope, approach, resources, and schedule of planned testing activities — affectionately called the test plan document. This document identifies test items, features to be tested, testing tasks, personnel arrangements, and risks from any contingencies.

Why Create a Test Plan?

Creating a test plan makes software testing a planned and organized software quality assurance activity. Without a plan, work becomes loose and arbitrary.

The Significance of a Test Plan

Test Planning Process

Testing Process Standards

• Testing models
• Traditional pyramid model
• Ice cream cone model
• Diamond model
• Testing process improvement

What Does a Test Plan Include?

• Human and time resource allocation
• Responsibility assignment
• Risk control

Testing Objectives

Product Quality Objectives

• Test whether the implemented product meets design requirements.
• Verify that the product’s specified operations are implemented and running stably.

Testing Activity Quality Objectives

• All test cases have been executed.
• All automated scripts have passed.
• All critical-level defects have been fixed.

Resource Configuration

Human Resources

• How many testers are needed
• What skills testers must possess
• Whether pre-job training is needed

Test Environment Resource Configuration

Hardware resources: servers, computers, phones, printers Software resources: operating systems for different platforms, database software, multiple browsers Network environment: what network environment to test in — intranet or internet Testing tools: which tools to use

Risk Control

Risk refers to unpredictable consequences, such as events, hazards, threats, and other special situations.

Objective risks:

• Objective factors — unavoidable risks:
• Not enough staff, and no suitable candidates can be found short-term
• Colleagues sick and on leave
• The development team cannot deliver code on schedule
• The environments, scripts, and data needed for the test environment have not been prepared; testing cannot proceed
• Risks cannot be fully controlled; only follow patterns to reduce the impact of risks.

How to Create a Test Plan

1. Task Received

• The test manager receives the software testing requirements and requirements specification.

2. Analyze the Testing Task

• Fully understand the requirements of the software under test.
• Evaluate the progress, status, complexity, and risk of the software under test.

3. Resource Planning

• Form the testing team.
• Prepare human resources.

4. Create the Test Plan

• Research and determine all aspects of the test plan.

5. Review the Test Plan

• The entire testing team participates in reviewing the test plan.

5W1H Method

What — Subject

• What to test
• What type of testing
• What are the characteristics of the software under test
• What is the test environment

When — Time

• When to start testing
• When to submit defect reports
• When to end testing

Why — Reason

• Why this test needs to be performed

Who — Participants

• Who the software is provided to
• Who executes the test cases

Where — Location

• Where software testing is performed
• At which step is testing considered complete

How — Method

• How to conduct testing
• How to write test case documents
• How to control risks

Plans are fixed, but people are flexible…

• Use purposeful goals to guide test progress planning, rather than planning for the sake of planning.
• Ensure your plan can be adjusted as changes occur.
• The test plan needs to be reviewed and executed by the entire testing team together.

Common Testing Tools

Test Management Tools (project process management)

• HP QC TD
• JIRA (international tool, most widely used, easy to use)
• ZenTao / Chandao (domestic tool)

Functional Testing Tools (automated script testing) (black-box)

• HP QTP (script recording tool, automating repetitive click work) — VBScript
• Selenium

Performance Testing Tools (black-box)

• HP LoadRunner (most widely used, easy, commercial software)
• Apache JMeter (free)

Code Testing Tools (white-box testing)

• JUnit

Open-source test management tools:

• Bugfree: BugFree is a Bug management system independently written using PHP+MySQL, referencing Microsoft’s R&D process and bug management concepts. It is simple, practical, free, and open source (following GNU GPL). The name “BugFree” has two meanings: first, the hope that bugs in software will decrease until there are none (Free); second, it is free and open source, available for anyone to use and distribute.
• Bugzilla: Bugzilla is an open-source defect tracking system (Bug-Tracking System) that can manage the entire lifecycle of defects in software development — submission (new), resolution (resolve), and closure (close). Bugzilla is an open-source Bug Tracking System designed specifically for Unix.
• TestLink: TestLink is a web-based test case management system. Its main functions include test case creation, management, and execution, as well as some simple statistical functions.
• Mantis: Mantis Bug Tracker (also known as MantisBT) is a lightweight open-source defect tracking system based on PHP technology, providing project management and defect tracking services via web interface. In terms of functionality and practicality, it is sufficient to manage and track small and medium-sized projects. More importantly, it is open source and requires no fees.

Open-source functional automated testing tools:

• Watir: Full name “Web Application Testing in Ruby,” pronounced like “water.” It is an automated functional testing tool based on web patterns.
• Selenium: Selenium is a tool for testing web applications. Selenium tests run directly in the browser, just like a real user operating it. Supported browsers include IE (7, 8, 9, 10, 11), Mozilla Firefox, Safari, Google Chrome, Opera, etc. Main features include: testing browser compatibility (whether your application works well across different browsers and operating systems), testing system functionality (creating regression tests to verify software functionality and user requirements), and supporting automatic recording of actions and automatic generation of test scripts in .Net, Java, Perl, and other languages.
• MaxQ: An open-source web functional testing tool.
• WebInject: WebInject is a free tool for automatically testing web applications and web services. It can be used to test various system components with HTTP interfaces (JSP, ASP, CGI, PHP, AJAX, servlets, HTML forms, XML/SOAP web services, REST, etc.), and can be used as a testing tool to create HTTP-level automated functional, acceptance, and regression tests. The tool lets you run many test cases and collect/report results, provides live result display, and can also be used to monitor system response times. WebInject can serve as a complete testing framework controlled by its own GUI, or optionally be used as a standalone test runner (a text/console application) that can be integrated and invoked from other test frameworks or applications.

Open-source performance automated testing tools:

• JMeter: Apache JMeter is a Java-based stress testing tool developed by the Apache organization. It was initially designed for web application testing but later extended to other testing areas. It can be used to test static and dynamic resources such as static files, Java servlets, CGI scripts, Java objects, databases, FTP servers, and more. JMeter can simulate heavy load against servers, networks, or objects to test their strength and analyze overall performance under different stress types. It can also perform functional/regression testing on applications by creating scripts with assertions to verify that your program returns the expected results — and for maximum flexibility, JMeter allows assertions to be built with regular expressions. You can use it for graphical performance analysis or to load-test your servers/scripts/objects under heavy concurrency.
• OpenSTA: OpenSTA is a free, open-source performance testing tool based on CORBA (Common Object Request Broker Architecture). It works by virtualizing a proxy server and using a specialized script control language to record all HTTP traffic passing through the proxy server. Test engineers analyze the performance metrics and HTTP data collected by OpenSTA’s performance indicator collector to evaluate the performance of the system under test.
• DBMonster: DBMonster is a Java open-source project that connects to databases via JDBC, so it can run on any platform that supports Java and JDBC. It was originally created to serve database developers, helping them generate large amounts of regular or irregular data for tuning their databases. DBMonster controls data-generation behavior through two XML files (a configuration file and a schema file): the configuration file specifies the database to connect to, the username and password for the connection, the schemas to operate on, retry counts, and other global settings, while the schema file specifies the rules for generating data for each field of each data table.
• Web Application Load Simulator: LoadSim is a web application load simulator that allows you to create simulations and run them against web servers.

Career Classifications in the Internet Industry

Salaries in the internet industry are relatively high — earning more than other industries after one year is normal. What positions are there in the internet industry, and which traditional industry backgrounds are they suited for?

1. Product Manager
2. UI Design (Photoshop / Illustrator / Premiere / After Effects)
3. Front-end Development (CSS / HTML / JS)
4. Back-end Development (Python / Go / Java / C++)
5. DBA (MySQL / Oracle / MongoDB / Redis)
6. Operations/DevOps (Linux)
7. Testing (software testing)
8. Algorithm Engineer
9. Big Data Engineer (Hadoop)
10. Android / iOS (mobile development)
11. Architect
12. Product/Growth Operations
13. Networking (routing & switching / hardware storage, etc.)
14. Security (penetration testing / web attack & defense / vulnerability scanning)