The Ultimate Competence Matrix for Performance Testers
How advanced are your skills as a performance tester? Do you know precisely where your strengths lie, what you still need to work on, and how to get the extra performance testing knowledge that you need?
Competence matrixes are a great way for you to instantly map your professional level of expertise - and see what you need to improve to advance on your career path. Until now, there’s only been one well-known matrix for programmers. Today, I’m happy to share this matrix specifically for performance testers.
WIthin five minutes, you’ll be able to pinpoint your level - from a business and technical perspective all the way through to your personal and communication skills. Once you’re able to identify which skills you’re missing, you’ll know which areas to focus on improving.
I recommend that you strive to maintain a balance between all of these different skills. Make sure that you don’t have a gap of more than two levels between the skill sets. For example: if you’re an expert when it comes to knowing the service infrastructure but a total beginner when it comes to working with workload models, you won’t know how to stress your infrastructure to find and fix the bottlenecks. Therefore, you should try to lessen the gap in your skill levels.
Finally, I strongly recommend paying special attention to your personal and communication skills - these are precious no matter what your profession is!
|Knowledge & Expertise||Area||Beginner||Intermediate||Advanced||Expert|
System Under Test
|Business Domain||Knows the general purpose of the service – but nothing about the expected load.||Has an in-depth knowledge of the service users and their activity. Knows the daily load profile and peak hours.||Tracks performance metrics in production to verify the testing results.||Knows business-priorities well. Informed on the business development plans.|
|Technical Architecture||Knows the subsystems list for the service.||Easily navigates the components inside the service's subsystems.||Knows internal object structure for the components, knows libraries and frameworks used.|
|Infrastructure||Knows the testing cluster structure in terms of hosts. Does not know the host’s configurations and network topology.||Knows the main configuration characteristics for testing and production servers||Knows the protocols and technologies used to interchange data between service components.||Knows network topology for testing and production. Knows load balancing configuration used.|
|Deployment and Operation||Relies fully on system administrators for deployment and configuration.||Can start and stop the service, as well as read and interpret the logs.||Easily navigates configuration options for the service and environment, and is able to change it if necessary||Can deploy and configure the service from the scratch.|
|Operating Systems||Uses the desktop OS and knows that the server OS exists and differs from the desktop one.||Knows the basics of the OS administration, desktop and server.||Knows multiprocessing principles in OS. Understands how virtualization works.||Knows perfectly how OS performs internally. Knows metrics that characterize this performance. Can do OS fine-tuning.|
|Networking||Knows how to ping servers and can tell the difference between WiFi and Ethernet.||Knows the difference between the router, network switch and hub. Knows what VLAN is - and understands the difference between TCP and UDP.||Perfectly navigates L3-L7 protocols. Has memorized the main TCP flags and states.||Knows network traffic routing principles and difference between unicast and multicast traffic.|
|Programming languages and technologies||No programming skills. Has a basic understanding of web-technologies (web server + browser).||Can do shell scripting and writes simple programs in C/Java/perl/python. Understands how compiler works.||Navigates popular design patterns and architectures (MVC, ORM).||Can develop multithreaded applications and work with TCP-sockets.|
|Data Storage and DBMS||Knows that there is special server role of data storage.||Familiar with at least one of the popular database management systems and can write simple queries.||Can install and configure various database servers.||Understands the internal architecture of the database and how its engine works. Can write complex queries and to optimize the structure of the data.|
Methods and Approaches
|Working with SLAs||Doesn't know what an SLA is||Understands the SLA set up by the customer. Able to interpret and compare test results to the SLA.||Knows common sense rules for response times. Can offer some SLA if customer has no estimation.|
|Workload Models||Thinks in terms of virtual users and scenarios. Creates tests by adding as many virtual users as possible.||Distinguishes between tests for measuring response times and tests aimed at measuring the system’s capacity.||Distinguishes between hit-based and scenario-based tools and open and closed workload models. Able to use them efficiently.||Makes spike-, recovery-, endurance-, exploratory-tests.|
|Localization of Performance Issues||Doesn’t know what the resource bottleneck is. Performs only requested tests.||Can detect the CPU/Disk/Network limit. Varies workloads to understand changes in service responses.||Can find misconfiguration issues for the service and OS. Performs exploratory tests to locate resource bottlenecks.||Can locate problematic locks, knows TSA (Thread State Analysis) and USE method (Use/Saturation/Errors.|
|Load Generators||Uses only aggregated information, taking it from the generator's results or calculating it manually. Stores them in bugtracker comments or in emails to colleagues.||Can build complex testing scenarios, parameterize the tests, and separate data feeding from the scenario.||Uses hit-based tools where applicable. Can test non-HTTP protocols.||Can write own generator implementation or plugin for testing non-standard protocols.|
|Building and storing load test reports||Uses only aggregated information, taking it from the generator's results or calculating it manually. Stores them in bugtracker comments or in emails to colleagues.||Uses graph plotting tools to see distribution and timeline data during the test. Stores results in the folder hierarchy locally or shares on the network.||Uses a fully automated results collector and storage function. Utilizes interactive online reports.||Uses storage with automated tests comparison feature and regression sequence analysis.|
|Resource monitoring tools||Peeks into the “top”/task manager during the test||Starts Dstat or Perfmon at the beginning of the test and simultaneously reviews them with the test graphs.||Integrates resource monitoring with the load generator. Resource utilization data is stored in the same place as the test results.||Sets up non-standard and business-related resource metrics.|
|Automation Tools||Prepares data and runs tests manually.||Writes shell-scripts to generate data and automate common tasks. Uses scheduled and delayed test runs if necessary.||Uses task automation tools like MSBuild, ant, and maven.||Uses CI-like system to launch tests automatically, including deployment, load generating and reporting.|
|Debugging Tools||Doesn’t use debugging tools.||Knows how to start a tracer/profiler/sniffer, collect data, and pass it to the developer/administrator for analysis.||Analyses data from tracer/profiler, and informs developers about localized problems. Uses Sniffer to diagnose network problems.||Know how to use debugger to analyse crash dumps. Can take stack trace dumps under load and interpret them.|
|Single Test||Examines basic aggregates: average, median, and standard deviations.||Considers distributions, percentiles and timeline graphs.||Uses pivot tables and graphs. Analyses request group slices. Analyses relations between resource monitoring and server responses.||Time taken for analysing single test is minimized.|
|Tests Comparison||Compares two tests by their cumulative response times||Compares two tests by their response time distributions||Compares request group segments to localize the source of fluctuation||Analyses sequence of similar tests by several aggregate parameters.|
|Communication Skills||Prefers email and bug tracker as communication channels. Problems are solved not very fast.||Has established connections with developers and administrators, thus enabling faster problem resolution.||Joins administrators and developers in one-to-one discussions on the architecture, infrastructure and development of the service.||Management invites to meetings on service roadmap. Initiates himself meetings on performance problems in the system.|
|Teamwork||Requires significant time from colleagues for help and training.||Is able to step in for colleagues when they are sick or on vacation.||Actively helps colleagues and juniors to investigate problems. Writes useful notes and articles on the knowledge base.||Leads the team. Manages a group of subordinate load testers.|
If you liked this matrix, I recommend printing it and putting it on your wall. You can check off areas every time you improve a skill and when you gain a full level-up to the “Advanced” or “Expert” status.
Feel like you need more training? Here are some resources that might help you:
Or share your questions and comments here!