Both network operators and infrastructure vendors need to test LTE networks under realistic conditions of subscriber demand. The advent of LTE introduces a number of new challenges for capacity testing and requires different techniques and expertise from those used for 3G networks. LTE has a flatter architecture than UMTS, which means that the baseband network interfaces traditionally used for capacity testing are no longer accessible in the LTE network. This article discusses these challenges and describes a method of capacity testing over the RF based on a well-proven air interface.
The traditional approach
Testing of mobile network infrastructure is traditionally performed in two stages – first in the laboratory and then in the field. The R&D testing in the lab takes the form of deep functional tests to ensure compliance to 3GPP specifications, first of all using a single user equipment (UE). Using the latest generation of test mobiles, several UEs can then be simulated to prove that the base station operates correctly when several devices are connected and to optimize the base station scheduling algorithms used to allocate resources to the UEs on the network. Testing with a purpose-built multi-UE test mobile typically gives repeatable and precise results, and allows the network to be tested before real handsets or mobile terminals are available, since handset development lags behind that of network infrastructure.
In the field the next stage is drive testing, but because few other users are connected to the network in the pre-deployment phase the drive test may not yield results that effectively simulate performance at full capacity. A purpose-built capacity test system is designed to generate high traffic loads to stress the end to end performance under a wide range of scenarios.
There will be situations in a real network where large numbers of subscribers wish to use services simultaneously, for example at an airport where 500 passengers disembarking from a transatlantic flight all switch on their phones at the same time; or simply in large cities where hundreds of thousands of people are using their handsets and data modems at the same time. The use of network capacity testing allows the network equipment vendor to test whether the network and base stations (eNodeB, or eNB) can handle situations like these reliably, and what maximum capacities can be specified before it becomes necessary to install additional equipment.
The LTE network comprises a number of different elements that include servers, the extended packet core (EPC), base station (eNodeB) and the antenna systems, as shown in Figure 1. It is important to be able to test the complete network under a range of realistic scenarios, for example when there are multiple cells, several thousand UEs connected, MIMO operation and users wanting various services.
Using capacity test systems
Both operators and network infrastructure vendors need to perform capacity tests before a network can be fully rolled out. Operators need to link their capacity test campaigns to their deployment plans to assess the end-to-end performance of vendor equipment – both for new networks and for upgrades of an existing network – and they need to be thoroughly confident of system behaviour and performance prior to deploying the live network.
Operators also require the ability to measure and compare the performance of their vendors’ equipment. Each vendor has a unique method for implementing the various scheduling algorithms and the differences between them will impact both performance and efficiency, and hence the end to end performance. Capacity testing allows an operator to compare the results obtained under a variety of different scenarios in a consistent manner.
Benefits to vendors
Infrastructure vendors can use capacity testing to optimize the performance of their equipment under real-life operating conditions, such as carrying out statistically-based testing with a large number of UEs and more cells, over an extended test duration. The ability to emulate real-world usage scenarios gives the vendor further assurance that performance in a live network will match their expectations.
The network vendors’ core R&D teams develop the eNodeB and other network components and then use functional test mobiles for 3GPP feature testing. This core functionality has passed detailed functional testing in R&D before being handed over to the capacity test teams, whose remit is to validate complete system behaviour in real world scenarios and ‘push the envelope’ – stretch the performance beyond standard tests for specification compliance.
As vendors increasingly move into a services business rather than simply supplying network hardware, they take more responsibility from start to finish for ensuring network performance and reliability, and they benefit from this improved ability to test in the lab. Functional testing of features cannot fully characterize how they will operate in a commercial network, with a large numbers of users and a range of other services being used in parallel, and the better the understanding of the capabilities at the limits of performance, the easier it is for the services part of the business to deliver and maintain reliable networks.
Time spent diagnosing why a particular feature is not working in the field reduces the efficiency of both the R&D and managed services teams and increases costs. Hence testing functionality and performance in a lab provides a working benchmark for field testing. Although network capacity testing cannot fix every problem, it can play a vital role in helping to improve the initial testing and characterization and then in providing a lab-based system for recreating any issues that have been observed in the field.
When problems do occur, the mobile operator needs quick and credible responses. An independent, commercially-recognized test system allows the vendor to provide detailed feedback on potential causes and resolutions.
Why LTE capacity test is different
3G operators have already experienced difficulties in coping with data-hungry smart phones that saturate the network and can prevent subscribers from making voice calls. LTE promotes an even richer and more diverse mix of subscriber services – including voice, video, web browsing, e-mail and social networking – that will have very different data traffic profiles and therefore impact the network performance in a way that is difficult to predict. This is much more difficult to test than for 2G networks like GSM, where the bulk of traffic is centred on voice and SMS.
At the same time, compared to UMTS, the LTE standard has removed the Radio Network Controller (RNC) that was a governing element in the UMTS Radio Access Network (UTRAN). This was traditionally used as the access point into the 3G network for high capacity testing, so it becomes necessary to test over the air interface. For 2G and 3G networks, some operators adopted the ‘wall of phones’ approach for end-to-end testing into the NodeB in the UTRAN, but due to the larger number of UEs per cell that LTE supports, along with the increased service complexity and feature churn, this becomes impractical.
Figure 2 shows a comparison of the UTRAN with the LTE network architecture. As mentioned above, the LTE network has been simplified compared to 3G UTRAN with the removal of the RNC node. This simplification of the network has concentrated more functionality into the eNodeB base station, which can now communicate directly with other eNBs to handle functionality such as device handover between base stations. A flatter network architecture leads to improved data latency (the transmission delay between the transmitter sending data and the receiver receiving it) and better support of delay-sensitive, interactive and real-time communications.
Comparison of 3G UTRAN and LTE radio network architectures
The removal of provision for circuit-switched voice services in LTE allows a move to all-IP based networks, with all voice traffic eventually being carried over IP. IP-based networks enable existing IP standards to be reused, simplify the network and reduce cost due to the substantial existing market in IP-based equipment. Because of this, Quality of Service (QoS) has become more important in LTE, not least because subscribers are becoming much more demanding of customer service and will not hesitate to switch provider if they are dissatisfied. Historically, IP networks have operated on a ‘best-effort’ basis, but this is not acceptable for real time voice and video services which have a limited tolerance to delay. The Quality of Service mechanism ensures that each service is prioritized correctly by the network, which compares it according to the traffic type with all the other traffic it is handling at any particular time.
LTE network capacity test
Gradually increasing the number of UEs in use makes it possible to simulate the traffic expected on a real network, including UE handover within and between base stations. When the number of UEs is in the hundreds a capacity test tool such as the Aeroflex EAST500 is the best solution for generating a test scenario based on realistic usage conditions.
Once R&D functional test has established that functionality is maintained in new releases through regression testing, the capacity test campaigns focus on tests relevant to operator customers. Tests are performed with different service mixes, increasing both the number of UEs and the number of cells in use. During this process, new functional releases will also be introduced that increase capability and these will be fed into capacity tests.
Once a satisfactory level of capacity capability has been established, then performance testing can start, with the aim of ensuring the maximum efficiency is being obtained from the network – in other words, the maximum number of bits transferred per dollar. It is the operator who determines the route to optimize the network through performance testing, since the key eNodeB algorithms that impact performance are proprietary. Complex multi-dimensional optimization is necessary to balance network efficiency, individual UE performance (Quality of Service) and application performance (Quality of Experience). Test campaigns may be customized to achieve operator differentiation, for example new features or higher data rates, and for changing the service mix or patterns of usage. They may also wish to capitalize on their experience from previous issues or their own performance criteria.
Capacity test challenges
Keeping up with 3GPP specification changes is a complex task. The specification version must be kept aligned between the suppliers of eNBs, network and all the tools, and furthermore, the specification change requests incorporated must also be aligned. Feature priorities can change frequently, driven by new operator customers or changes in operator requirements, additional features are often added to the network when equipment vendors win new contracts.
Test equipment maturity can be a significant issue. The amount of effort required to achieve interoperability with the network is often underestimated. This can result in suppliers needing to spend many months working on-site while at the same time continuing to develop their equipment. The priority for both operators and network vendors is to spend their time debugging and optimizing systems rather than the test tools.
It can also be a significant challenge to stay up-to-date with 3GPP while aggressively increasing the number of UEs, and in practice, it is an arduous task to increase the number of UEs and improve stability while still maintaining functionality. If the UE simulator provider supports multiple 3GPP versions, this can make the transition between different versions of the specification easier to plan and with a substantial engineering support team, the UE simulator provider can react quickly to changes in functional requirements.
Test equipment limitations
Problems arise when field issues cannot easily be recreated in the lab. Lab testing has the advantage of being cheaper and more repeatable than field testing and it is possible to create scenarios that do not often occur in the field or to load the network before there are real subscribers to do so. However, unless the lab test scenarios are accurate, there is a risk of missing problems that are then only discovered when customers begin to complain. One of the key areas to examine is the RF environment which a UE simulator is able to recreate in the lab.
Without advance knowledge of the problems that may occur, it is difficult to determine the UE simulator measurement data required to identify them – strong measurement and diagnostic tools are, therefore, essential to help engineers attack problems in a logical and structured manner.
Vendors each develop their own algorithms based on many years of experience of doing so. Capacity testing not only provides an insight into the level of performance and efficiency of these algorithms, but can also be a useful tool for the vendors’ engineers to investigate ways to further improve them.
UE simulator selection criteria
It is advantageous if the same test equipment can be used both in the lab and over the air in the field, as it is then more likely that problems found in the field can be reproduced in the lab.
Rapid root cause identification is also critical. Most UE simulators focus upon the KPIs required by operators, but for effective fault finding and performance optimization UE stack measurements are also required from all levels in the stack.
Algorithms that are proprietary to the eNB, such as the scheduler, directly impact performance and efficiency, so it is important to ask how well the data provided gives visibility of what the scheduler is doing – detailed low level data such as resource block usage is essential.
Importance of complete test
Most operators and vendors consider that using a UE simulator over RF is an acceptable compromise compared to using real handset UE devices. However, UE simulators without RF – where a different baseband interface is used – stretch the analogy between real handsets and the UE simulator too far. The EAST500 is an integrated system for LTE network capacity testing over RF with real data services and applications and has been designed to provide capacity test that simulates the real world in a lab environment. The EAST500 incorporates an evolution of the industry-standard LTE RF test mobile interface for eNB development, the Aeroflex TM500, which has already completed interoperability testing with every major infrastructure vendor worldwide. This large installed base enables operators and vendors to dramatically reduce the time taken by manufacturers to get their capacity test system up and running compared to alternative systems.
The EAST500 has also been designed to ensure that the test lab is able to recreate both real-world scenarios and field issues and to provide a comprehensive set of KPI measurements along with the ability to use lab equipment over the air in a real field environment, to compare results with high levels of accuracy. Measurements are made throughout the UE stack to give the best quality diagnostic information.
The hardware is flexible and scalable depending upon the number of UEs and number of cells required. The system offers true end-to-end LTE load testing over RF in both FDD (frequency division duplex) and TDD (time division duplex) environments. The system roadmaps address the most challenging 3GPP functionality requirements and with work already started on 4x4 MIMO and LTE-Advanced, they are prepared for the challenges of yet higher data rates.