Chapter 2: Monitoring in ADMADE

A Stakeholder Analysis of Monitoring Information in ADMADE
Data Sets
Information Flow
Use of Maps
Assessing Wildlife Populations
Quota Setting
Evaluating Data Quality
Percentage of Field Patrols Recorded

A Stakeholder Analysis of Monitoring Information in ADMADE

RURAL COMMUNITIES

In terms of information needs, rural communities require data upon which to manage their wildlife resources, such as making quota recommendations, planning anti-poaching operations, and ensuring that all hunting fees and regulations are adhered to. Whereas in the early years of ADMADE many of these chores fell almost exclusively upon the Unit Leader and his field staff, the new ADMADE structures dictate that more and more of these responsibilities lie with the various community management committees . Information available for management activities includes indicators of wildlife population trends (e.g., hunting statistics, observations on field patrols), field patrol results, poacher profiles, and Wildlife Conservation Revolving Fund (WCRF) statements.

Planning and implementing community development is at least complex, if not more so, than managing wildlife. In order to prioritize development needs, communities need information about household level food security, livelihood strategies, human population growth and distribution, health and education services, wealth distribution, and intra-community dynamics.

Catching and preventing mismanagement of money and other project resources is another important need of rural communities. Whether it is ammunition and rations taken on field patrols, or the amount of money received by the Community Resource Board (CRB) for an entire year, accountability and transparency of resource use is critical for program success. To ensure accountability, communities need information on field patrol supplies, license sales, animals hunted, community development projects, and Unit expenditure summaries. When mismanagement does occur, communities need a monitoring system that is responsive enough that it will catch the problem at an early stage so that corrective measures can be taken.

Under the 1998 Zambia Wildlife Act, Community Resource Boards will also be required to develop comprehensive resource co-management agreements between themselves, the relevant government agencies, and private industry. Negotiating a co-management agreement is an information-intensive activity in itself, requiring recent resource inventories and baseline data on management operations and resource use. In addition to helping develop co-management agreements, resource monitoring will also be an important component of all co-management plans.

Communities will face other information needs when reviewing and renegotiating safari hunting concessions with safari operators. One of the major determinants of success of ADMADE in a GMA is the performance and integrity of the safari operator and his professional hunters. Monitoring data can be used to evaluate the past performance of a safari operator, assess the economic potential of a hunting block, and negotiate new concession fees.

Local land-use plans have been developed for most of the ADMADE Units in the Luangwa Valley, and will be developed for the remaining areas in the next year or two. Land-use plans are developed in participatory workshops, and are broad-spectrum, comprehensive sets of proposed actions designed to resolve and prevent land-use conflicts. Resolutions from a land-use plan may include shifting human activities away from wildlife areas, implementing a new project such as an electric fence or road rehabilitation to address community needs, or clarification on the roles of the various actors in ADMADE. Developing a land-use plan is a complex, participatory exercise, which requires monitoring data such as wildlife habitat needs, safari hunting trends, Unit demography, community development priorities, and revenue flows.

ZWA UNIT STAFF

In addition to conducting management operations, Unit staff has an interest in ADMADE's monitoring system in a way that no other stakeholder has: they are the source of most of the data. Village and regular scouts, under the leadership of the Unit leader and his deputies, collect all of the safari hunting, field patrol, and poacher arrest data, and are recorders for other types of data such as crop damage and snaring pressure. The scouts and their supervisors need to know not only the results of their monitoring work, but also feedback on their methodology of data collection. One of the on-going efforts by extension staff from Nyamaluma has been to increase the capacity of Units to collect, store, and analyze the various forms of monitoring data.

Interviews conducted for this study revealed that providing evidence for judicial proceedings is another common use of monitoring data at the Unit level. Poacher case records and field patrol dataforms become important pieces of evidence when poachers are brought to court. Likewise when scouts are accused of an offense, such as improperly confiscating property, or injuring or killing a poacher, dataforms from the operation may be used in adjudicating the case.

At a slightly higher organizational level, Wardens, who are responsible for an entire command (Zambia is broken into nine commands), have their own information needs. Wardens are in charge of all personnel matters, allocation of human and material resources, and monitoring wildlife populations in their command. Commands also get a percentage of safari hunting revenues for their operations, so they have a vested interest in ensuring that safari hunting is being managed properly and sustainably. Some commands also have biologists on staff, who typically have few resources to work with and may rely heavily on data from scouts in ADMADE areas. Biologists may also assist Unit staff in monitoring activities, such as data analysis or organizing ground transects.

NYAMALUMA INSTITUTE

Nyamaluma's information needs are as diverse as the roles it plays. To fulfil its function as a training institute, Nyamaluma requires information about staff numbers, retention rates, educational background, and training needs. In its role as a source of extension services, Nyamaluma needs all the same information as communities and Unit staff. Likewise, as the primary liaison between communities and ZWA headquarters, international donors, and the safari industry, Nyamaluma requires the same type of information as these other stakeholders.

Nyamaluma is able to fulfill so many roles partly because it functions as the central nervous system of ADMADE's monitoring system. There are very few monitoring activities in ADMADE that were not designed, initiated, and continuously supported by the staff and technical resources at Nyamaluma. In exchange for offering technical support with data analysis and presentation, Nyamaluma gets a copy of all monitoring data which it enters in a master database.

ZAMBIA WILDLIFE AUTHORITY HEADQUARTERS

As far as ADMADE is concerned, senior officers in ZWA want to know how successfully wildlife is being conserved in the project areas, and how communities are benefiting from the program. On a more immediate level, they need information on staffing issues and supplies for field operations. At the policy and strategic planning levels, they need to know how government and industry practices affect the success of safari hunting and ADMADE, and how those policies might be altered or supplemented with new initiatives. The decision to adopt ADMADE as the official government wildlife management policy for GMAs was based in part on information meticulously kept during the pilot Lupande Development Project. Similarly, the future evolution of wildlife management in Zambia will be based in large part on the monitoring results and experiences of ADMADE.

In 1999 an ADMADE Coordinating Office was opened at ZWA headquarters Chilanga. This office allows ADMADE to develop a badly needed presence in the day-to-day activities of the department. The coordinating office also provides field support to the ADMADE Units surrounding Kafue National Park, and liases with other government departments and the donor/NGO communities in Lusaka. The information needs of the coordinating office parallel those of Nyamaluma, and there is close coordination between the two branches. The coordinating office does not presently play a role in data processing and analysis, but once its future is stabilized monitoring is likely to become a larger component of its operations.

USAID

One of the challenges all donor funded projects must face are shifts in the donor's information needs. In the mid and late 80's, when ADMADE's funding agreement was developed and approved, USAID's reporting and evaluation frameworks were generally oriented to measuring the impact of individual projects. Furthermore, biodiversity conservation was a goal in itself. In the mid-90's, USAID as an agency became more 'results oriented', reflecting a larger movement in the US federal government to improve accountability and effectiveness. Oversea missions were told to develop strategic plans for the country, and streamline their project portfolios to be more coherent and integrated around a hierarchical framework of goals and objectives.

As a result of this shift, the last phase of USAID funding for ADMADE under the current project agreement falls under Strategic Objective One: To increase the rural income of selected groups. Under this strategic objective, and its three intermediate results, a variety of performance indicators are listed for which ADMADE must provide data in its quarterly reports. These indicators include the net income of rural households, access to finance, value of commodities marketed, improved land and labor productivity, and the number of clients of support institutions. ADMADE, with its strong programmatic emphasis on wildlife conservation, does not fit neatly into this new branch of USAID's strategic objectives chart, and has had to strengthen its data collection in several areas. To measure performance towards USAID's objectives, ADMADE needs to report the number of people benefiting from community development projects, the nature of those benefits, the effectiveness and efficiency of management activities, and variables which impact the long-term sustainability of the program. This translates into improving data collection on revenue flows, community awareness and support for the program, impact of community development projects, management capacity at the local level, wildlife population trends, and performance of the safari hunting industry.

Because donors are not involved in day to day management, USAID for the most part only requires aggregated summaries of monitoring data, not all the details. Furthermore, because ADMADE's impact monitoring data is compiled with data from other supported projects, USAID perfers quantitative over qualitative data, and absolute values instead of simply relative measures or trends. They also require data which is representative of the project as a whole, instead of just selected areas, to ensure that the results are a valid measure of ADMADE's overall performance.

WILDLIFE CONSERVATION SOCIETY (NEW YORK)

SAFARI INDUSTRY

Most operators and professional hunters in the safari business are motivated as much from a passion for wildlife and hunting as the financial rewards. They have an interest in ensuring that hunting in Zambia is managed profitably and sustainably, and by extension are interested in all data that is used to guide management of wildlife. More specifically, they are interested in any information that can be used for setting hunting quotas, to ensure maximum profit without jeopardizing the success of future hunting seasons. They also don't want to be in the position of advertising wildlife trophies that don't exist, because this can ruin one's reputation in a fairly fickle market. When competing for concessions, safari operators need data upon which to base their bid for the hunting block. This includes measures of wildlife abundance, past hunting success, management capacity, and characteristics of the local communities.

Both safari operators and professional hunters must recruit foreign hunters to hunt in their area. Much of this marketing takes place during the annual Safari Club International Convention in Las Vegas. To market their hunting block to wealthy, sophisticated, and demanding clientele, safari operators need to present evidence of the status of wildlife and hunting success in their area. To a lesser but growing extent, safari hunters are also interested in the conservation benefits of hunting, and desire information about the sustainability and ethics of hunting in a certain area. ADMADE's 'Green Bullet' certification program is one of the newer elements of its monitoring program which provides prospective hunters with this type of information.

MINISTRY OF ENVIRONMENT AND NATURAL RESOURCES, ENVIRONMENTAL SUPPORT PROGRAMME

EINMS has the mandate of compiling a directory of all environmental data in the country and developing institutional partnerships to facilitate exchange and enable cross-sectorial analyses. Wildlife, along with forest resources, fisheries, and clean air and water, is one of the key resources identified by the ESP to be of national significance and at risk. ADMADE has one of the most complete datasets on wildlife in the country, particularly outside national parks in the game management areas where much of the wildlife resides. Furthermore, from a methodological standpoint, the EINMS and its institutional partners have a lot to learn from ADMADE's ten year experience of using community residents in natural resource data collection, and conducting analysis with state of the art computer applications.

The CEMP program is similar to ADMADE in that it strives to involve communities in the management of natural resources. Two of the four CEMP pilot areas actually overlap ADMADE units. However CEMP differs slightly from ADMADE in that it focuses on a larger suite of resources, including forest products, fisheries, and mining, and is being implemented through district level government. Nevertheless, national, district, and community leaders in the CEMP program could all benefit from ADMADE's monitoring system, both in sharing content and methodology. Because CEMP is likely to evolve into a national program, ADMADE areas may potentially gain as well, in strengthening the capacity to manage non-wildlife resources, diversifying the resource base for community development, and working more closely with non-wildlife departments in local government.

ENVIRONMENTAL COUNCIL OF ZAMBIA, WILDLIFE RESOURCES MONITORING UNIT

NGO COMMUNITY

Information Flow

When discussing information flow, it is useful to categorize datasets based on their origin. ADMADE's core datasets can be broken down as follows:

As can be seen, many of the datasets, including most of the really interesting ones, are collected at the community level. With the exception of household demography, which is an infrequent door-to-door survey done by a contracted community member, all community-based monitoring data is collected by ADMADE scouts. This is not accidental, as there are many advantages of using scouts for data collection. Most scouts spend significant amounts of time in the field, and are in a good position to record the types of phenomenon of interest. Scouts fall under a command and control system, increasing the effectiveness and efficiency of data collection, supervision, and storage. Most scouts are local residents, which may increase interest and confidence in the data by the community at large. Finally, all village scouts go through training at Nyamaluma at one time or another, where monitoring skills have been built into the curriculum.

In the future, it is hoped that a broader spectrum of community members will become involved in monitoring, particularly members of the different management committees. However at least in the data collection aspect of monitoring, it seems likely and appropriate that village scouts will remain the backbone for the foreseeable future.

Although it is impossible to generalize, Figure 2 below illustrates the general flow of information and some of the common barriers and bottlenecks.

Figure 2 - Information Flow, Bottlenecks, and Interventions in ADMADE
Click on the image for a larger picture

Use of Maps

Map production was a strong emphasis at Nyamaluma during the early days of ADMADE. The research team digitizing dozens of Survey Department maps in order to produce smaller scale base maps of each area. The basic layers for fall field maps include the Unit boundary, streams and roads (important for patrol navigation), scout camps, and an overlay of 5 km2 grids. Most areas also have a map layer for human settlements, and many have additional layers for water holes, safari camps, project sites, and VAG boundaries.

Nyamaluma produces two types of maps. Letter sized base maps are designed primarily for navigation and locating grid numbers during field operations. Base maps are printed in bulk, and distributed to all village scouts involved with monitoring. Custom made flipchart-sized wall maps are designed to present monitoring summaries during group meetings. Wall maps show results such as the grids where safari hunting is active, field patrol coverage, and areas of land-use conflicts.

Although it is difficult to quantify, the maps distributed by Nyamaluma have undoubtedly proven invaluable for planning and evaluating field operations. Because they are customized for individual areas and printed in bulk, base maps are available to scouts for field patrols and safari monitoring. The large wall maps make monitoring results relatively easy to understand for community residents. ADMADE areas are quite large, and without an integrated capacity to produce maps, ADMADE would have found it much more difficult to visualize patterns in management and land-use within a Unit and across larger areas. Maps have also have an educational value in helping to convey the finiteness of a management area. This is an important realization for community residents who may never have seen all corners of their GMA and perceive it to be without end.

Recently, scouts have also starting to use base maps for recording monitoring data (see Chapter 4). Using a simple system of writing tick-marks in the appropriate grids and keeping a separate base map for each type of observation, scouts will be able to maintain their own spatial summaries of where they have gone and what they have seen. By reducing the dependency on the extension staff from Nyamaluma, and reducing the amount of lag time for analysis, it is expected that scouts and community leadership will make more frequent use of monitoring data when planning activities.

Assessing Wildlife Populations

Measuring Wildlife Populations: The Choices
Counting wildlife is a field of science on its own, and there is a multitude of methods. Counting techniques can be broken into two broad categories, direct and indirect. Direct methods count animals themselves, even though the whole population might not be counted. Indirect methods measure the signs animals leave behind, such as tracks or scat, or some other secondary phenomenon that is related to the total population, such as a sex ratio or harvest statistics.

*methods used by ADMADE

Both direct and indirect methods can suffer from problems with measurement or sampling. Measurement problems include inconsistent measurement (e.g., different equipment, different observers, different scales or techniques), lack of sufficient precision in the measurement, and inaccurate measurement. Sampling problems include insufficient sample size, temporal bias (e.g., measuring a 'snapshot' which may not reflect the true state of the population if measured over a larger spatial or temporal scale), and non-representative selection of observations.

Constraints with Direct Methods
Unfortunately, the direct methods of counting wildlife although generally more valid are not particularly suitable for ADMADE areas. Aerial surveys in savanna woodlands are only reliable at detecting the very large conspicuous mammals, such as elephant, buffalo, large antelope, etc. Small mammals and animals which are secretive - such as the cats - or well-camouflaged are not easily counted from an airplane. Aerial surveys, as well as ground transects, are also very sensitive to factors such as time of day, experience of observers, time of the year, fire, etc. which can bias results by unknown magnitudes.

A second problem with aerial surveys is the cost. ADMADE areas spend about $5/km2 on field management per year (NPWS, 1998), and aerial surveys in Zambia cost about $0.80 - $1 /km2 inclusive at a 10% sampling rate (Jachmann, per. comm, 1998). Spending 20% of the annual management budget on a single survey is simply not a feasible option for any protected area.

Other challenges with aerial surveys include obtaining the equipment and technical expertise to conduct the operation. Hence it seems unlikely that rural communities will ever have the resources to conduct or even contract aerial surveys themselves, and it will continue to be a donor or department initiated activity. Thus while aerial survey data will always be an important supplement to other types of data, it will most likely never be repeated frequently enough or free from bias to be the definitive data source for wildlife populations in ADMADE GMAs.

Ground transects, either on foot or by vehicle, have their own disadvantages. Vehicle transects in the heavily-wooded, lightly-roaded ADMADE GMAs are vulnerable to highly biased results because only animals that are within visual distance of the road can be counted. Foot transects on the other hard can reach areas off the beaten track, but are even more expensive to conduct than aerial surveys because of the greater number of people required, longer amounts of time, and logistical support needed (Jachmann, personal communication, 1998). These methods also require skilled biologists and computer software for analysis, and like aerial surveys are susceptible to even small changes in methodology. Thus while ground transects can also tell a part of the story, they alone are not a panacea for measuring wildlife.

Indirect Methods
If the direct methods of counting wildlife seemed plagued with problems, indirect methods do not fare much better. Out of the possible choices for indirect methods, the only techniques which are really feasible in ADMADE areas are collecting hunting statistics (e.g., trophy size, search effort, hunting success) and interviewing key informants. Other indirect methods such as capture-recapture studies, counting tracks or dung piles, or recording responses to baits (e.g., carcasses, animal vocalizations) require advanced scientific expertise and equipment beyond the capacity of Unit staff.

Unlike direct methods, most indirect methods only produce an index or proxy for the wildlife population, which generally can not be used to calculate abundance or density. Data from these indicators is only comparable if (1) we assume that even though the sampling is not random, it is at least consistent over the time frame of interest, and (2) we get enough observations that any bias from fluctuations in sampling average out. However if measured consistently over time and with enough samples, an index can in theory detect change in a population.

Like all techniques, indirect methods are sensitive to changes in sampling procedures or the way measurements are taken. Indirect methods such as hunting statistics and key informant interviews are particularly vulnerable to unknown bias from sampling. Safari hunters do not randomly select animals from a population, and scouts do not randomly select the areas they patrol. Because of this non-random sampling, it is difficult to make many inferences about the population as a whole.

Temporal Bias
One advantage of the indirect methods used by ADMADE, hunting statistics and key informants, is that there is much less temporal bias. Hunting statistics are collected over a six-month hunting season, and key informants are asked to recall their last 3-5 years of observations. Hence any short-term fluctuations in the population tend to average out. In contrast, aerial surveys and ground transects are done much less frequently, and the status of the vegetation and the state of the observers on the day of the count has a much greater, though still unknown, influence on the results.

Animal Movements
Another challenge of counting wildlife in GMAs is animal movements. Most of the ADMADE GMAs, and all of the GMAs richest with wildlife, border national parks. Although many of the details of animal migration are not well known, there are definitely large movements of wildlife across these borders. Animal migrations in and out of GMAs may be triggered by rainfall patterns, over-browsing, predator pressure, or anthropogenic disturbances such as hunting pressure or habitat loss. In the absence of accurate information on the magnitude and timing of these movements, the preferred censusing methods would be those that measure a sufficiently large enough spatial and time scale to not be heavily influenced by these movements. These would favor large scale aerial surveys or combining indirect monitoring data from multiple adjacent GMAs with sufficient samples throughout the year.

Confidence Limits
It has been said that the most dangerous data is not the data which is inaccurate, but inaccurate data which is not known to be inaccurate. An important characteristic of data quality is some kind of built-in measure of the accuracy or confidence we have in the data. Direct methods are at an advantage to indirect methods in this aspect. Because direct methods are directly measuring the variable of interest, well-established statistical formulas can be applied to calculate a confidence range. The problem with indirect methods are that the observations and the actual phenomenon of interest are related by some third mechanism or chain of events, which is often complex and not well known. Hence with indirect methods, it is usually not possible to quantify how accurate our results are, although we can usually qualitatively guess the likely direction of error and perhaps its magnitude. Confidence in indirect methods can also be evaluated by comparing multiple indicators (see below).

Sustainability
Another factor, outside of science, that must be considered when planning a monitoring strategy in the context of CBNRM program is sustainability and the degree of community participation. A common characteristic of wildlife monitoring in conservation programs all over Africa is an almost complete reliance on outside financial and technical resources. If these outside resources can be sustained indefinitely, (e.g., a protected area that operates from an endowment, a financially secure wildlife department), then basing monitoring around the relatively expensive direct methods would be favored. However if the support from outside resources has a known termination date, then the monitoring program should at least include a component that can be realistically sustained with the resources of local management. Large scale direct methods will always have an important role for collecting baseline data, developing management plans, prioritizing conservation areas, etc. However unless CBNRM projects also establish within their monitoring systems techniques that can be sustained by field staff after donor pull out, they are jeopardizing the long-term likelihood of achieving the conservation goals.

Table 2 below compares the most common methods for counting wildlife in Zambian GMAs.

Table 2 - Selected comparison of wildlife counting methods

Comparing different measures of wildlife populations is always good practice, however it is particularly important for indirect indicators. Indirect methods tend to be more susceptible to external, unmeasurable influences, which we hope will average out given enough measurements. Indirect methods also generally suffer from unknown confidence limits, so comparing different indicators is the one of the few ways to assess whether the results can be trusted. When several indicators are in agreement, then the likelihood that they reflect an actual pattern in the population is enhanced, which is essentially mimics the purpose of a confidence interval.

The second strategy for using indicator data is to avoid the Nosnibor effect, drawing only conclusions at levels which the data can support. For wildlife population monitoring, there are basically three types of conclusions that can be drawn: (1) presence/absence of a species, (2) increasing/decreasing population trend, and (3) an estimate of absolute abundance or density. As you increase the level of conclusion sought, the data requirements become more and more rigorous. For example the data requirements for estimating absolute abundance are more demanding than those for detecting a change in the population, which in turn requires more stringent data than determining whether a species is present or absent. ADMADE indicator data can be used at most to establish the presence/absence of a species or estimate a population trend. The conclusions are much stronger on the absence/presence question than population trends, and indicator data can not be used to measure abundance at all. Likewise, aerial census and transect survey data, where it exists, can be used to estimate abundance, but is more dependable for supporting conclusions about the population trend of a species.

One of the indirect indicators used by ADMADE, observations of key informant, contribute not only by supporting or contradicting the results of other indicators, but also by interpreting the meaning of other, perhaps more statistically robust, measurements. For example, detecting that a species is declining is an important finding from quantitative monitoring methods, but determining whether that species is declining due to disease, migration, or over hunting can only be answered by key informants or other more detailed studies.

Data from direct and indirect methods can support each other not only in cross-checking conclusions, but also in designing sampling protocols. For example, an important part of planning aerial or ground transects is deciding how to stratify the area based on vegetation or topological zones. Sampling units that have a more homogeneous population distribution result in narrower confidence limits and more robust conclusions about the population. Monitoring data from key informants, such as scouts, professional hunters, and trackers, can assist dividing the GMA into appropriate census areas. Likewise, existing monitoring data can also suggest the total area used by migratory species, so that surveys avoid missing part of the population.

Quota Setting

• multiple stakeholders with varying interests
• developed in a context of complex ecological and socio-economic systems
• the roles and power levels of the different stakeholders reflect larger institutional and policy structures

There are two general models for setting quotas: percent off-take and trial-and-error. These models can be mixed and matched, but in any particular quota setting exercise one approach is likely to dominate the other. In the percent off-take method, information about the reproductive biology of a species is used to calculate a maximum percent of the population that can be harvested each year. In the trial-and-error method, (1) a guess-estimate is made of a reasonable quota, (2) the population is rigorously monitored to detect upward or downward trends, and (3) the quota is updated on a regular basis to reflect new information as it becomes available.

In practice there is actually a third model of quota setting, which becomes the default when there is no monitoring data to support either of the other two models. In this method, the interests of different stakeholders are reviewed and a quota is negotiated based on the power relationships of the various parties. Unfortunately this method rarely produces a sustainable quota yet becomes the norm when there is no monitoring system in place. This is largely the system that was used in GMAs before ADMADE was established by the government.

Which method of quota setting is most appropriate is largely a function of the type of monitoring data available. Calculating the percent off-take requires a fairly accurate estimate of the total population. This may be possible on fenced game ranches, but is rarely practical in the open areas of Zambia. As an example, the table below shows the results of a series of aerial surveys in Munyamadzi GMA.

Table 3 - Aerial Survey Results, Munyamadzi GMA
(Jachmann, 1994; Jachmann, 1996; Jachman 1998)

In ADMADE areas, which generally lack recent population estimates even as broad as the ones above, the only alternative for setting quotas is the trial-and-error method. According to this model, the population must be rigorously monitored and the quota reviewed and adjusted on a regular basis. In ADMADE, this review is made at the end of each hunting season. The indicators that are used to monitor the wildlife populations include:

HUNTING STATISTICS

Assuming that safari hunters are generally selecting trophy animals in the same way from year to year , trophy size is a fairly direct and valid index of the age of the oldest adult males in the population. This by itself is enough of a reason to use trophy size as an indicator, as sustainability of the program requires managing for trophy animals. In addition, changes in trophy size also reflect changes in age structure in the population, which in turn is an indicator of the birth-death rate, and therefore population growth or decline. Thus with a sufficiently large sample size to reduce sampling error, trophy size is a fairly valid measure of the population.

Hunting success. Hunting success is the percentage of safari hunters seeking a particular species that successfully found and shot an animal. Before each hunt begins, the village scout accompanying the client is supposed to ask which animals he is looking for during the hunt. At the end of the hunt, the animals actually taken are compared with what was desired to calculate hunting success.

Hunting success intuitive seems like a good index for the population, however it can be biased by several factors. Firstly, it is assumed that the amount of time spent looking for an animal is roughly uniform across years, or at least a consistent spread. A low hunting success from ten classical safaris (where hunters can spend more than a week hunting) would be a more significant result than a low hunting success from ten mini safaris (less than seven days hunting). Currently ADMADE doesn't weight or breakdown hunting success data according to length of the hunt, however the assumption is that these variations will be similar from year to year.

The validity of hunting success as an indictor of population change is also very dependent on sample size. A hunting success of 33% would be interpreted much differently if was a result of one out of three hunters finding the animal than if it resulted from eight out of 24 hunters taking an animal.

Hunting effort. Hunting effort is defined as the number of days it takes a safari client to find and shoot an animal. When hunting effort increases, it implies that animals are more difficult to find, presumably because there are fewer of them. Hence hunting effort is also intuitively a good indicator of population change. However like the other indicators, the validity hunting effort as a proxy for population change is dependent on other confounding factors being controlled or at least averaged out.

Other factors which affect hunting effort include the time of the season, because it takes longer to find animals during the beginning of the hunting season when grasses are high. The time it takes to find and shoot an animal may also vary depending on whether the hunter is on a classical safari, where he has more time to be selective, or a mini-safari, where all hunting must end after seven days. Finally, hunting effort can vary greatly according to the style of the professional hunter, and other species on the hunt which may necessitate hunting animals in a particular order.

With so many confounding factors, hunting effort is one of the weaker indicators of population change. In theory, the effect of factors other than population size should cancel out with a sufficiently large sample size. Nevertheless when taken in concert with other indicators, hunting effort can help detect changes in population.

Number of animals hunted. The total number of animals which have been legally hunted is an indication of the offtake from hunting. By itself, the number of animals successfully hunted my not be very meaningful, as it depends on a variety of factors (such as the number of hunters who wanted an animal). However when matched with other trends from other indicators, the trend is more likely to be valid. Number of animals hunted is also important because it represents the sample size for hunting effort and trophy size, and helps the magnitude of changes in the quota.

KEY INFORMANTS

However like any other indicator, using key informants can result in bias or error. There are a multitude of possible confounds in using observational data, all well described in any social science research methods text. Some of the more relevant problems for ADMADE include influences on responses from other scouts, bias resulting from limited field experience or limited to certain times or in certain areas, recall error, mistaken observations, and hidden agendas.

The basic strategies for making informant data more valid are the same strategies for making any other kind of data valid: measuring the data has to be as objective as possible, and confounds have to be controlled for. Collecting data from informants (i.e., interviewing them or administering a questionnaire) is always a challenge due to the number of possible influences from the interview situation. In the past, ADMADE has not used any systematic or controlled method for interviewing informants; their opinions on wildlife were solicited during group meetings such as quota setting exercises. This generally resulted in forming an opinion by group consensus, with one or two scouts speaking for the others, and perhaps suppressing dissenting views. In 1999, Nyamaluma introduced a more objective way of interviewing informants, by developing a "Population Trend Survey" questionnaire which is designed to be administered to key informants individually and under controlled circumstances. This approach offers promising possibilities, because it should now be possible to look for agreement between scouts and the same scout over time to study the validity of informant observations.

Informants involved during the quota setting process include:

Scouts. Village and civil servant scouts form the backbone of ADMADE's law enforcement and monitoring programs in the field. On both field patrols and safari monitoring their primary mission is to go where wildlife (and poachers) is likely to be found. Hence they can offer some of the most well-informed observations about populations. The amount of time in the bush scouts spend in the bush is not well known, however it is certain to vary widely. A 1998 ADMADE report estimated that scouts patrol an average of 20-40 days per year. This is probably an underestimate, but to what extent is not known. One thing certain is that due to lack of transport many patrols are restricted to the immediate area around the camp.

Professional hunter and tracker. The job of a professional hunter is to guide safari clients to wildlife. Professional hunters also have good vehicles to move around the area and often many years of experience in an area. Hence they are usually a good source of information on wildlife population dynamics. Most professional hunters also employ a local resident with extensive experience in the bush to be a tracker, providing another good source of information.

Unit leader. Although due to the nature of his job the Unit leader doesn't go out on patrol as much as his scouts, he nevertheless is a valued source of information on wildlife. The Unit leader is in frequent contact with his scouts, and serves as a focal point for all other wildlife issues, such as reports of poaching activity, crop damage, legal hunting by Zambians, and disturbances to habitat. A good Unit leader has a grasp on the main wildlife problems in his area, and when combined with other informants can provide valuable insight into population trends.

Ex-poachers, honey gatherers, etc. Other informants who generally have extensive knowledge of the bush include local residents such as former poachers, honey gatherers, firewood gatherers, etc. Their background, bush experience, and other agendas may be less well understood than scouts or professional hunters, however they provide corroborating evidence for population trends and can offer important insights into resource use patterns.

FIELD PATROL DATA

INDICATOR AVAILABILITY AND AGREEMENT

The coefficient of agreement is the percentage of all possible pairs of indicators that are in agreement. Higher values indicate that indicators tended to agree with each other, while lower values indicate there was disagreement between the indicators. Note however that when the 0's (no-conclusive indicators) are removed from the analysis, the coefficient of agreement increases significantly. This implies that you rarely have indicators that are in opposite directions (e.g., one indicating a negative trend and another indicating a positive trend). Agreement also suggests that the indicators are valid measures of the same underlying phenomenon (i.e., population change). This result and the number of indicators available supports the conclusion that ADMADE's use of indicators for monitoring wildlife population trends is based on sound methodology.

Table 4 - Indicator Data Available for Community Quota Setting

QUOTA SETTING MEETINGS

Meetings usually can be completed in one day, although sometimes a second day is required if the area contains more than one hunting block or the meeting got off to a slow start. These quota setting meetings were introduced program wide in 1997 and most meetings have been facilitated by staff members from Nyamaluma. However it is hoped that in the future the Unit staff and community leadership will play a larger role in conducting these meetings.

The basic strategy for wildlife assessment in quota setting is to compile data from as many sources as possible and look for agreement between indicators. Before the meeting begins, the facilitators and Unit staff prepare all the hunting statistics for the current season, and look for trends from previous years. A trend is defined as three or more year's worth of data suggesting a definite change in the population. The results of each indicator are summarized on a flipchart with the following symbols: + positive trend, - negative trend, 0 no discernable trend. Next to these are additional columns for the opinions of scouts, tracker, and professional hunter, which are filled in during the meeting.

Interpreting indicators is based on scientific principles, but also involves a very fluid and qualitative discussion. In most cases, the statistical indicators and the opinions of the scouts and Professional hunter are in agreement, and there is little debate. In some cases, the hunting statistics are self-contradictory, inconclusive, or unavailable, and more discussion is required between the 'human indicators'. In rare cases, the quantitative indicators contradict the observations of scouts and hunters. In general, the opinions of people outweigh the statistical measures. From my observations of four quota setting exercises in 1998, the indicators which carry the most weight during group discussions are in order of their importance:

1. opinion of the professional hunter
2. opinions of the tracker and the scouts
3. opinions of others at the meeting (e.g., ex-poachers)
4. hunting statistics

Conservatism is part of ADMADE's overall design strategy in quota setting. When data is missing or inconclusive, the quota tends to remain the same. Even when data indicates that a population is increasing, quotas are more often than not adjusted slowly. My observations were that although there was usually someone at the meeting who wanted to drastically increase the number of animals on quota, the final consensus was much more conservative. This dampering effect of individual voices is another important benefit of broad participation in community meetings. Both reproductive rate and home range requirements are sometimes taken into account when adjusting hunting quotas. Even if all indicators are positive, quotas for species such as leopard that reproduce slowly and are thinly spread are likely to be incremented by only 1 or possibly 2 animals. Some species, such as lion, have a separate quota for male and female species.

Once the quota setting meeting is complete, the results of the discussion are copied from the flipcharts and onto the Quota Setting Worksheet, which is then signed by all those present as well as the chief. This paper is then sent to ZWA headquarters for the annual review meeting of the national quota setting committee.

The facilitation role of Nyamaluma's extension staff is an important element in community quota setting exercises. In addition to the technical knowledge they bring, along with summaries of previous monitoring data and logistical support such as a vehicle and flipchart materials, they also represent an outside 3rd party which is perceived to be objective and may be needed to bridge differences between stakeholders. Whether ADMADE Units will be able and willing to conduct quota setting exercises with the same level of professionalism without support from Nyamaluma remains to be seen.

Evaluating Data Quality

Ensuring data quality is a concern for any monitoring program. Anxieties about data quality increase the further one gets away from the source of the data. For example, a bureaucrat in Lusaka will be less able to detect possible biases and inaccuracies in a summary of ADMADE's monitoring data than a Unit leader. Data quality is difficult to measure, and errors can either multiply or average out as data becomes more and more aggregated. Evaluating data quality is challenging unless there is an independently measured standard to compare with. Unfortunately CBNRM programs like ADMADE rarely have the benefit of an independent standard, because there are no other projects or studies recording this kind of data in these areas.

Although measuring quality data objectively is difficult to quantify, it is relatively easy to take steps to control the quality of data. ADMADE uses a mix of procedural controls and quantitative tests to increase the reliability of its data. Although no system for data quality assurance can be made foolproof, these controls are reasonable precautions given the available resources.

DATA QUALITY CONTROLS - PROCEDURAL & HUMAN

Only certified scouts are supposed to be selected for safari monitoring or appointed as the data recorder on field patrols or investigating crop damage. In reality, some non-certified scouts, including civil servant scouts, may also wind up recording data recording on field patrols or safari hunts. The Unit leader or his designated deputy have the responsibility to weed out those scouts who don't show competence in monitoring, and over time, only those scouts who have the ability and interest to record data wind up as monitors.

No financial incentives. To reduce the likelihood that dataforms will be falsified, village scouts are given no material incentives for recording data. Instead it is hoped that scouts will be motivated from an understanding and appreciation of the benefits of collecting data. Although the policy of not providing incentives for the extra work is unpopular among scouts, and may have other disadvantages as well, it has most likely achieved its objective of minimizing falsified data and there have been no known cases were dataforms were purposefully fabricated.

Dataform certification. The first line of defense against bad data comes at the field level. Each dataform is supposed to be reviewed and certified by the Unit leader or his appointed deputy soon after the data is collected . Certifying data forms in the field can catch omitted responses on forms, as well as detect certain irregularities and outliers. In practice, the degree to which dataforms are certified depends in large part on the individual unit leader or deputy assigned to monitoring, and frequency of contact with scouts.

Spot checking during data entry. The data entry staff at Nyamaluma have a lot of experience entering and analyzing data, and have a good feel for what is and what is not a reasonable measurement. Many mistakes can be caught during the data entry process, including problems with inconsistent units and outliers, for example a hippo shot in the hills. Data which is suspect is not entered into the database, and common dataform mistakes are noted in preparation for the next training on monitoring.

Interpreting analyses. Previewing the results of an analysis can also highlight errors in data, Most of the extension staff from Nyamaluma who spend a good bit of time in the field have a pretty good intuitive feel for the major problems and accomplishments in different areas. When summaries or graphs depict results that seem counter-intuitive, the discrepancy may be traced either to an incorrect analysis, error in data processing, or bad data.

DATA QUALITY CONTROLS - COMPUTER/QUANTITATIVE

Field and table validation rules. In addition to enforcing the integrity of linkages between related tables, the new database also has the ability to validate all data being entered against preset validation rules. For example, the date a hunt ended can not come before the date it started (an error which in fact was encountered in the old system because the spreadsheet wasn't formatted to display the year of a given date). Similarly, table definitions specify which fields must have data, and which fields are optional. Table validation rules also prevent duplicate records, for example there can not be two field patrol observations entered for the same phenomenon in the same grid on the same day. Other validation checks are done programmatically during the data entry process, such as the check for valid trophy measurements based on the species hunted.

Statistical measures of data quality. Once data has gotten through field certification, data entry, and finally made it into the "system", it can still be examined for data quality. One of the advantages of using a well-designed database is that quantitative summaries and graphs can be easily and quickly produced. The following are actual examples of charts, maps, and tables of monitoring data that are built-in to the new ADMADE database and can be used to highlight data quality concerns.

Sample Size
An important and easily measured component of data quality is sample size. Summaries which are based on only a small number of observations are less likely to accurately reflect the population than those with a larger size. Very few observations in ADMADE's monitoring system are based on a random sample, so sufficient sample size becomes all the more important to minimize the bias introduced by sampling.
Fortunately sample size is easy to present to the user in tabular and graphical summaries. Figure 3 to the right is one of the many interactive charts in the new database and depicts the hunting success for hartebeest in all GMAs from 1994 to 1998. The blue diamond markers represent the hunting success (calculated as the percentage of hunters who shot a hartebeest out of those who stated they desired one at the start of their hunt), and should be read using the scale on the left. The square green markers represent sample size and should be read using the scale on the right.

Figure 3 - Hunting success of hartebeest 1994-98

Dispersion
The amount of dispersion in a set of measurements can suggest whether the data has been measured and collected properly. Histograms can quickly present the type of distribution curve of the sample data, which are expected to fit certain norms. Figure 4 below shows a histogram of trophy measurements for Cape Buffalo for all hunting blocks and all years combined. This fairly normal distribution is what we would probably expect from a natural population of trophy specimens, and suggests both that scouts are making measurements properly and that individuals are probably being selected from the population in a consistent manner.

Figure 4 - Histogram of trophy size measurements reveals
an expected distribution curve in measurements

Figure 5 below shows the number of day on patrol for two camps in the Chifunda Unit for 1998. A few observations are immediately apparent from this graph. First of all, Kanusha camp did almost no patrolling during the months of February to May - the rainy season. Hence any data from the scouts in that camp on poaching levels, animal abundance, or other phenomenon are likely to be biased by the lack of patrolling during this period. Secondly, there are no patrols recorded for the months of November and December. This can only be attributed to (1) there were no patrols during those months, (2) not all data has been entered into the database. Assuming the later, we also note that any summary of patrolling effort for this Unit for the year will likely under-represent the actual number of days patrolled.

Figure 5 - Number of days on patrol for Kanusha and Luelo camps, 1998

Figure 6 - timeline of safari hunting in Chanjuzi hunting block, 1998

We note from this map that not surprisingly scouts patrol more heavily around their base camps. While that realization may have implications on its own, it also must be considered when interpreting other results from field patrol monitoring. For example all of the safari hunting in Mumbwa GMA is done on the western side of the GMA, so at least for this year it would probably not be appropriate to use field patrol observations to examine the competition between safari hunters and poachers for the same animals. This spatial sampling bias also suggests that scouts may be impacting poaching activity near local settlements on the eastern side of the GMA, but are not patrolling areas closer to the park, where commercial poachers may be attracted.

Figure 7 - Map showing location of field patrols in Mumbwa Unit, 1997

DATA QUALITY CONTROLS - SUMMARY

Table 5 - Potential error and data quality controls

Percentage of Field Patrols Recorded

Scouts in interviews for this study unanimously stated that 100% of all field patrols are recorded on dataforms. However after reviewing the field patrols summaries it seems unlikely that all field patrol data actually makes it through the information chain to Nyamaluma. Not only do the results suggest low patrolling effort (20-40 days per year per scout according to a 1998 Nyamaluma report based on selected areas with 'good' datasets), but also some Units have lengthy gaps where no patrolling is recorded. Hence it is suspected that either the percentage of patrols recorded is less than 100% to begin with, or that dataforms get lost somewhere along the information pipeline to Nyamaluma.

Our visit to Mumbwa GMA afforded an opportunity for the first time to empirically study the percentage of field patrols recorded on dataforms. The Mumbwa Unit headquarter at Nalusanga maintains a Field Operations Record Book, a ledger for all field operations that originate from Nalusanga camp. Although the ledger book records just a subset of the fields of information on the ADMADE field patrol data forms, it does provide an independent record of field patrols.

I compared the field patrol records in the Field Operations record book with ADMADE Field Patrol dataforms for 1997 and 1998. For 1997, there are 17 field patrols which originated from Nalusanga recorded in Nyamaluma's database, and 16 original dataforms at the Unit Headquarters. However the Field Operations Ledger recorded 44 field operations during 1997, excluding operations such as escorts and check points. Thus for that year only 39% of the field operations were recorded on dataforms that reached the unit headquarters and Nyamaluma.

Reviewing the field patrol dataforms from 1997 reveals that all recorded patrols were between July and December of that year. Hence the most likely explanation for this low reporting percentage is that the first six months of data was either never collected or lost. However neither the Unit staff nor research staff at Nyamaluma had any memory of what may have caused this gap, and the missing data still results in a gross underestimate of patrolling and monitoring effort.

It should also be noted that Nalusanga camp lies on the border of Mumbwa GMA and Kafue National Park, and that scouts patrol in both areas. KNP is not an ADMADE area, and there is no base map for it. Hence although scouts are supposed to record patrols made in either area, it may not be entirely surprising that not all patrols, particularly day patrols, in the park are recorded.

For 1998, there were 39 field patrols originating from Nalusanga in the Nyamaluma database, this time spread almost evenly throughout the year. However the Nalusanga Field Operations ledger recorded 115 field operations, again excluding investigations, official escorts, funeral drills, etc. Thus only 32% of field operations were actually recorded for that year. The missing data from the ledger book indicates an unrecorded 122 patrols days and 421 man-days of effort. Thus for 1998, the number of recorded patrols from Nalusanga was only 32% of the actual, only 56% of patrol-days, and 67% of total man-days.

Of the non-recorded field patrols, 69 were in Kafue National Park and nine were in Mumbwa GMA. This pattern suggests the probable cause of the underreporting - scouts apparently don't fill in dataforms for all patrols in Kafue, even though they do submit dataforms for some of the patrols there. However another element of the problem is from non-recorded operations in the GMA. The nine patrols not recorded in the GMA represent 163 man-days of patrolling effort and monitoring data missing.

Many of the patrols not recorded were day patrols, confirming another suspicion that day patrols are widely underreported in general. Scouts or Unit staff may not want to use their limited supply of dataforms on day patrols, as running out of data forms has been a recurring problem in several areas. Or scouts may think that only patrols which warrant filling in a data form are those where a poacher is arrested, supplies are used, or something interesting is found. Most scouts who record data on patrols stated that they take notes on plain paper, and then fill in the dataform on their return.

It seems unlikely that the missing data was caused by a chronic problem of dataforms getting lost or destroyed after being filled out. Nalusanga has a spacious office and one of the more organizing filing systems we saw. Furthermore, the operations under question originated right from the headquarters, which is right on the main road so travelling to and from town does not require the standard relay of bush transport. In fact one would suspect that the percentage of field patrols recorded would be much higher in a camp such as Nalusanga, where scouts are based at the same camp where dataforms are distributed, reviewed, and filed. Hence the most likely explanation is that due to misperception or lack of supplies dataforms for many field patrols were never filled out in the first place.

These results should not be considered indicative of all ADMADE areas, because they reflect the rather special circumstances of only one camp in one Unit. However they do underscore the reality that the completeness of field patrol data is difficult to assess and should not be assumed to be 100%. One intervention by Nyamaluma that may help evaluate data completeness in the future is the 1999 Self-Appraisal dataform. If used properly, this dataform will provide future researchers an independent record of field operations at the camp level, as well as other time allocation categories such as construction, going for salaries, education programs, rest, etc.

Acknowledgements | About the Report | Acronyms | Introduction | Monitoring in ADMADE | Synthesizing Results | Interventions | Conclusion | Bibliography | Monitoring Framework | ADM Menu System | Data Analysis Conceptual Framework | Monitoring Workshop Notes | Additional Research Readers' Comments | Make Comments