Canadian Science Advisory Secretariat (CSAS) Proceedings Series 2016/004 Pacific Region

Canadian Science Advisory Secretariat (CSAS)
Proceedings Series 2016/004
Pacific Region
Proceedings of the Pacific regional peer review of a Simulation Modelling Tool to
Evaluate Alternative Fishery Closure Network Designs for Shallow-water Benthic
Invertebrates in British Columbia
October 23-24, 2013
Nanaimo, BC
Chairperson: Chris Pearce
Editor: Chris Pearce
Fisheries and Oceans Canada
Science Branch
3190 Hammond Bay Road
Nanaimo, BC V9T 6N7
February 2016
Foreword
The purpose of these Proceedings is to document the activities and key discussions of the
meeting. The Proceedings may include research recommendations, uncertainties, and the
rationale for decisions made during the meeting. Proceedings may also document when data,
analyses or interpretations were reviewed and rejected on scientific grounds, including the
reason(s) for rejection. As such, interpretations and opinions presented in this report individually
may be factually incorrect or misleading, but are included to record as faithfully as possible what
was considered at the meeting. No statements are to be taken as reflecting the conclusions of
the meeting unless they are clearly identified as such. Moreover, further review may result in a
change of conclusions where additional information was identified as relevant to the topics
being considered, but not available in the timeframe of the meeting. In the rare case when there
are formal dissenting views, these are also archived as Annexes to the Proceedings.
Published by:
Fisheries and Oceans Canada
Canadian Science Advisory Secretariat
200 Kent Street
Ottawa ON K1A 0E6
http://www.dfo-mpo.gc.ca/csas-sccs/
csas-sccs@dfo-mpo.gc.ca
© Her Majesty the Queen in Right of Canada, 2016
ISSN 1701-1280
Correct citation for this publication:
DFO. 2016. Proceedings of the Pacific regional peer review of a Simulation Modelling Tool to
Evaluate Alternative Fishery Closure Network Designs for Shallow-water Benthic
Invertebrates in British Columbia; October 23-24, 2013. DFO Can. Sci. Advis. Sec. Proceed.
Ser. 2016/004.
Table of Contents
SUMMARY ................................................................................................................................... iv
SOMMAIRE .................................................................................................................................. v
INTRODUCTION .......................................................................................................................... 1
PRESENTATION OF WORKING PAPER .................................................................................... 2
POINTS OF CLARIFICATION ............................................................................................... 2
WRITTEN REVIEWS AND COMMITTEE DISCUSSIONS ........................................................... 2
REVIEWER COMMENTS and QUESTIONS ......................................................................... 3
COMMITTEE COMMENTS and QUESTIONS ...................................................................... 5
CONCLUSIONS AND RECOMMENDATIONS ............................................................................. 7
ACKNOWLEDGEMENTS ............................................................................................................. 9
REFERENCES ............................................................................................................................. 9
APPENDIX A: AGENDA ............................................................................................................. 10
APPENDIX B: PARTICIPANTS .................................................................................................. 12
APPENDIX C: TERMS OF REFERENCE .................................................................................. 13
APPENDIX D: ABSTRACT FROM WORKING PAPER .............................................................. 15
APPENDIX E: WRITTEN REVIEWS .......................................................................................... 16
REVIEWER 1 ....................................................................................................................... 16
REVIEWER 2 ....................................................................................................................... 18
REVIEWER 3 ....................................................................................................................... 21
iii
SUMMARY
The use of fishery closures is a component of the management strategy for the commercial sea
cucumber (Parastichopus californicus) fishery (DFO 2011, DFO 2012). Of the many benefits
that may be derived from reserves, the primary purpose for existing fishery closures is to protect
a portion of the P. californicus population as a safeguard against potential overfishing, given
uncertainties in the current understanding of biology and population dynamics.
In British Columbia, a total of 13 areas where commercial sea cucumber fishing is prohibited
have been in place since 2008 (DFO 2012). To date, the number, size, and location of these
fishery closures have been determined using a set of arbitrary criteria developed from in-house
knowledge and expertise, and includes representativeness of surrounding harvested areas,
ease of definition for management purposes, size appropriate for monitoring, and an arbitrary
target of including 20 percent of harvestable shoreline within these fishery closures (Duprey et
al. 2011). Science advice was requested by Fisheries Management to provide a scientificallysound and transparent process for the development of a coast-wide network of fishery closures
for shallow-water benthic invertebrates.
The primary goal of this work was to develop a simulation-modelling approach to evaluate
alternative fishery closure network designs and fishery management scenarios for
commercially-harvested, low-mobility benthic invertebrates, with a particular focus on P.
californicus, using specified performance indicators. Simulation results were evaluated for their
potential to provide advice on size, spacing, and configuration of fishery closures in areas with
limited data (e.g. on species distribution, habitat suitability, and larval dispersal patterns). It is
intended that the simulation tool will be flexible enough to allow its application to other fisheries
for low- or no-mobility benthic invertebrate species for which fishery closures or other
management measures may be defined. The specific outputs of this work will also inform
current work that is underway to establish a network of MPAs in British Columbia, as per the
draft Canada-BC MPA Network Strategy and ongoing marine spatial planning processes, such
as the Pacific North Coast Integrated Management Area (PNCIMA).
These Proceedings summarize the relevant discussions and key conclusions that resulted from
a DFO Canadian Science Advisory Secretariat (CSAS) Regional Peer Review (RPR) meeting
on October 23–24, 2013, at the Pacific Biological Station in Nanaimo, British Columbia. A
Working Paper focusing on the simulation tool was presented for peer review.
In-person participation included DFO Science and Fisheries Management staff as well as invited
representatives from the Province of British Columbia, First Nations, environmental groups, and
the commercial fishing sector.
The conclusions and advice resulting from this review are summarized in a Science Advisory
Report along with the findings of the Research Document, both of which will be made publicly
available on the Canadian Science Advisory Secretariat (CSAS) website.
iv
Compte rendu de l'examen par les pairs régional du Pacifique sur l’outil de
modélisation pour simulation visant à évaluer différents réseaux de zones
fermées à la pêche des invertébrés benthiques des eaux peu profondes en
Colombie-Britannique
SOMMAIRE
La fermeture de la pêche fait partie de la stratégie de gestion de la pêche commerciale de
l’holothurie (Parastichopus californicus) (MPO 2011, MPO 2012). Compte tenu de l’incertitude
des connaissances actuelles en biologie et en dynamique des populations de P. californicus,
parmi les nombreux avantages qu’offrent les réserves, la fermeture de la pêche permet
principalement de protéger une partie de la population de l’espèce pour parer aux effets d’une
surpêche éventuelle.
En Colombie-Britannique, on compte 13 zones où la pêche commerciale de l’holothurie est
interdite depuis 2008 (MPO 2012). Jusqu’à présent, on a déterminé le nombre, l’étendue et
l’emplacement de ces zones fermées à la pêche à l’aide de critères qui restent arbitraires, bien
qu’ils s’appuient sur des connaissances et des compétences techniques internes. Ces critères
comprennent les suivants : les zones de fermeture doivent être représentatives des zones de
pêche environnantes, elles doivent être faciles à définir à des fins de gestion, et elles doivent
être d’une étendue appropriée pour pouvoir en assurer la surveillance. On a également établi
une cible arbitraire d’inclusion de 20 % de la côte exploitable dans la zone de fermeture de la
pêche (Duprey et al. 2011). La Gestion des pêches a demandé un avis scientifique pour mettre
au point, selon un processus scientifique et transparent, un réseau de zones fermées à la
pêche des invertébrés benthiques des eaux peu profondes sur l’ensemble de la côte.
Le but principal était de mettre au point une méthode de modélisation pour simulation visant à
évaluer différents concepts de réseaux de zones fermées à la pêche et scénarios de gestion de
pêche pour la pêche commerciale d’invertébrés benthiques peu mobiles (en particulier
P. californicus), en utilisant des indicateurs de rendement précis. Les résultats des simulations
ont été analysés pour déterminer leur potentiel à fournir des renseignements sur l’étendue,
l’espacement et la configuration idéaux des zones fermées à la pêche dans les régions pour
lesquelles il existe peu de données sur la répartition de l’espèce, la qualité de l’habitat, les
modèles de dispersion des larves, etc. L’outil de simulation doit être assez souple pour être
appliqué à la pêche d’autres espèces d’invertébrés benthiques peu mobiles ou non mobiles,
auxquelles on pourrait appliquer des fermetures de la pêche ou d’autres mesures de gestion.
Les résultats de cette étude pourraient contribuer aux travaux en cours visant à établir un
réseau d’aires marines protégées en Colombie-Britannique, en vertu de la version préliminaire
de la Stratégie Canada – Colombie-Britannique pour le réseau d’aires marines protégées et des
processus de planification spatiale marine en cours, tels que la zone de gestion intégrée de la
côte nord du Pacifique (ZGICNP).
Le présent compte rendu résume les discussions pertinentes et les principales conclusions
découlant de la réunion d’examen par les pairs du Secrétariat canadien de consultation
scientifique de Pêches et Océans Canada (MPO), qui a eu lieu les 23 et 24 octobre 2013 à la
Station biologique du Pacifique de Nanaimo, en Colombie-Britannique. Un document de travail
portant sur l’outil de simulation a été présenté aux fins d’examen par les pairs.
Les participants en personne comprenaient des employés des secteurs des Sciences et de la
Gestion des pêches du MPO ainsi que des représentants invités de la Province de la ColombieBritannique, de Premières Nations, de groupes environnementaux et du secteur de la pêche
commerciale.
v
Les conclusions et les avis découlant de cet examen sont résumés dans un avis scientifique,
tout comme les conclusions du document de recherche. Le tout sera publié sur le site Web du
Secrétariat canadien de consultation scientifique.
vi
INTRODUCTION
A Fisheries and Oceans Canada (DFO) Canadian Science Advisory Secretariat (CSAS)
Regional Peer Review (RPR) meeting was held October 23–24, 2013 at the Pacific Biological
Station in Nanaimo, British Columbia to review a Working Paper on simulation modelling tools to
evaluate alternative fishery closure network designs for shallow-water benthic invertebrates in
British Columbia.
The Terms of Reference (TOR) for the science advice (Appendix C) were developed in
response to a request for advice from DFO Fisheries and Aquaculture Management Branch
(FAM). Notifications of the science review and conditions for participation were sent to various
representatives with relevant expertise in the subject area, including internal (DFO Science,
FAM, and Oceans) and external (Province of British Columbia, First Nations, environmental
groups, and the commercial fishing sector) representatives.
The following Working Paper was prepared and made available to meeting participants prior to
the meeting:
Duprey, N.M.T., Curtis, J., Finney, J., Hand, C.M. Simulation modelling tools to evaluate
alternative fishery closure area network designs for shallow-water benthic invertebrates in
British Columbia. CSAP Working Paper 2013/P63.
The meeting Chair, Chris Pearce, welcomed participants, reviewed the role of CSAS in the
provision of peer-reviewed advice, and gave a general overview of the CSAS process. The
Chair discussed the role of participants, the purpose of the various RPR publications [Science
Advisory Report (SAR), Proceedings, and Research Document], and the definition and process
around achieving consensus decisions and advice. Everyone was invited to participate fully in
the discussion and to contribute knowledge to the process, with the goal of delivering
scientifically defensible conclusions and advice. It was confirmed with participants that all had
received copies of the TOR and draft Working Paper.
The Chair reviewed the Agenda (Appendix A) and the TOR (Appendix C) for the meeting,
highlighting the objectives and identifying the Rapporteur (Dan Curtis, DFO). The Chair then
reviewed the ground rules and process for exchange, reminding participants that the meeting
was a science review and not a consultation.
Members were reminded that everyone at the meeting had equal standing as participants and
that they were expected to contribute to the review process if they had information or questions
relevant to the paper being discussed. In total, 36 people (including two reviewers) participated
in the RPR (Appendix B).
Participants were informed that Matthew Slater (Department for Marine Aquaculture, imare Institute for Marine Resources, Bremerhaven, Germany), Stephen Smith (Bedford Institute of
Oceanography, Fisheries and Oceans Canada, Halifax, Nova Scotia), and Ilona NaujokaitisLewis (Department of Ecology and Evolutionary Biology, University of Toronto, Toronto,
Ontario) had been asked before the meeting to provide detailed written reviews for the Working
Paper to assist everyone attending the peer-review meeting. Participants were provided with
copies of the written reviews. Paul Ryall (Director, Resource Management, Pacific Region) gave
a presentation on Day 2 of the meeting to explain the rationale behind the request for science
advice on the topic.
The conclusions and advice resulting from this review will be provided in the form of a Research
Document and a SAR providing advice to FAM on simulation modelling tools to evaluate
1
alternative fishery closure network designs for shallow-water benthic invertebrates in British
Columbia.
The Science Advisory Report and supporting Research Document will be made publicly
available on the Canadian Science Advisory Secretariat (CSAS) website.
PRESENTATION OF WORKING PAPER
Three of the co-authors (Nicholas Duprey, Jessica Finney, and Janelle Curtis) gave a
presentation that was broken down into the four major sub-models of the simulation tool: habitat
suitability, metapopulation dynamics, dispersal, and fisheries management. The abstract of the
Working Paper is given in Appendix D.
POINTS OF CLARIFICATION
TOR
There were no questions regarding Objectives 1–4 and 6 of the TOR (see Appendix C). In
regards to Objective 5 of the TOR, one of the Committee members wanted to know what was
meant by the term “proxies”. The authors responded that this term indicates other potential
methods that could be used for designing fisheries closure area networks.
Working Paper Presentation
There were only minor points of clarification; most being questions/concerns dealing with the
paper itself, not the actual presentation. They were as follows:
1. Question: Was the research data on abundance stratified?
Authors’ response: Yes, at 2-km intervals.
2. Question: Was the parameter space used representative of area?
Authors’ response: Not sure. Need to test this.
3. Question: Is all the data used to build the model?
Authors’ response: Twenty-five percent were used to build the model and the rest were used
to test the model and displayed in the predicted vs. observed densities.
4. Question: Were data based on quadrats or scaled up to transect?
Authors’ response: Scaled up to transect, using the Hajas method.
WRITTEN REVIEWS AND COMMITTEE DISCUSSIONS
In advance of the meeting, written reviews were solicited from three individuals who are
knowledgeable in the area: Matthew Slater (imare - Institute for Marine Resources), Stephen
Smith (Fisheries and Oceans Canada), and Ilona Naujokaitis-Lewis (University of Toronto). All
three reviewers felt the paper was very well written and their full reviews are given in Appendix
E.
The reviewers felt that the overall model was comprehensive (given the available data), well
thought out, and would be a useful tool for fishery managers for managing stocks of fished, lowmobility, benthic invertebrates. They all commended the authors on a well-developed simulation
modelling tool. Two of the reviewers (Slater and Smith) joined the meeting via teleconference to
present their reviews in person. The Chair read the review of Naujokaitis-Lewis.
2
REVIEWER COMMENTS AND QUESTIONS
One of the reviewers wondered why there was no optimal scenario from the model runs. The
authors pointed out that the “optimal scenario” was meant to be addressed with the sensitivity
analyses. There actually wasn’t an ideal design because almost all simulations fell below the
limit reference point (LRP). No changes to the Working Paper were requested by the
Committee.
A reviewer disagreed with the assumption of importance of temperature in the habitat-suitability
model, given the small temperature ranges provided in Table 2-1, as the cited literature was
primarily related to temperature extremes well beyond the normal range for the species
investigated. They suggested that the authors should investigate the growth and behavioural
data in Zamora’s publications on temperature for better references and more nuanced
information on temperature effects in temperate species. The authors agreed to do this.
One of the reviewers asked whether the density model would benefit from ground-truthing
predicted densities in non-surveyed areas post-modelling. The authors responded that they
already have some survey data collected that would address this, but more would be needed.
This would be for possible future runs of the model and not for the present Working Paper.
It was felt that the survival rate for g23 individuals may be low (should be 0.3–0.4, instead of the
predicted 0.2). The authors noted that they could look at sizes/ages and survival rates from
other studies to possibly predict survival of various size classes better. The authors pointed out,
however, that the model output was not very sensitive to changes in survival rate. The
Committee agreed with the authors’ justification and no changes to the Working Paper were
requested.
One of the reviewers commented that the fecundity estimates seemed extremely high and
wondered if some spawning and/or hatchery attempts had been made with this species. The
authors replied that there was minimal data on fecundity of P. californicus and noted that the
fecundity estimates and egg viability were used as back-of-the envelope calculations to ensure
that the matrix model was close, not as an actual parameter in the model. The reviewer
admitted that the fecundity prediction method was good and no changes to the Working Paper
were requested.
In regards to tree complexity (tc) of the habitat and density models, it was mentioned that the
Working Paper would benefit from a graphical representation of what a tc=3 (or 7) tree would
look like. The authors agreed to add this to the Working Paper. Subsequent to the meeting, one
of the authors of the Working Paper pointed out to the Chair of the meeting that there are many
citations that people can consult that would explain tree complexity and argued that a graphical
presentation of a tc=3 tree is not needed in the Working Paper. The Chair agreed and the
addition was not made to the document.
One of the reviewers asked how strongly correlated the bag fraction (bf) was with the tc
parameter. The authors commented that this relationship wasn’t examined in the Working
Paper, but that they had an array with all of the outputs. So, it would be possible to examine this
relationship. No changes to the Working Paper were requested, however.
It was felt that the document could benefit from more discussion surrounding the authors’ choice
of boosted regression tree (BRT) over other ensemble methods such as bagging, random
forest, or MAXENT or a consensus model (based on a combination of a range of distribution
model methods). The authors commented that they have used multiple approaches in other
studies, which produced different outputs. They also noted that, even with other ensemble
methods, there would still be uncertainty in the model outcomes and that is why it’s important to
incorporate different habitat suitability maps. They chose the BRT technique, however, due to a
3
reference which said it was better than other analyses and which used BRT in a similar study to
the present one. The authors agreed to add more discussion on the topic in the Working Paper.
Correlated predictor variables are common in these kinds of studies. One reviewer wanted to
know what the correlated variables were and how the results would be affected if you took one
or more of these correlated variables out of the analysis. The authors admitted that it would be
valuable to have a table with variables that were used in each run, showing degree and type of
correlation, with some clarifying text. They agreed to add this to the Working Paper. Subsequent
to the meeting, one of the authors pointed out to the Chair of the meeting that the table would
not really be beneficial and that the clarifying text added to the Working Paper should be
sufficient. The Chair agreed and the addition was not made to the document.
One of the reviewers requested clarification on the depth used in the model as various depths
(15, 30, 50, 650 m) are mentioned in the paper at various points. The authors clarified that
survey data typically only goes to 15 m, but the model assumes a depth of 30 m as densities
are unlikely to change from 15 to 30 m and it is unlikely that removal activity occurs at depths
greater than 30 m. The authors also noted that depth was removed for some runs of simulations
and there was no difference when depth was removed. The authors agreed to add some
clarifying text in the Working Paper as to the choice of depth and agreed to see if there was
enough data to run the analysis when depth was restricted to 15 m.
One of the reviewers wondered what the results of the simulation’s BRT analysis were saying
with respect to the relationship between initial abundance and percent decline. Why is there an
increase in percent decline for populations greater than 45 million? The authors remarked that
you start off by estimating Total Allowable Catch (TAC) in an area. Each subsequent fishing
event has the same fixed TAC removed (since areas are rarely re-surveyed to re-set the TAC).
So for higher populations, more individuals are removed each year. The authors agreed to
clarify this better in the text of the Working Paper.
It was suggested that additional performance metrics be included in the Working Paper,
specifically the % deviance explained by the model (a measure of the goodness of fit between
the observed and modelled values), the area under the receiver operating curve [AUC, a
threshold independent metric that combines the trade-off between sensitivity (the true positive
proportion) against the false positive proportion], and the true skill statistic (which accounts for
both omission and commission errors, and ranges from −1 to +1, where +1 indicates perfect
agreement and 0 random fit). The authors agreed to present % deviance and AUC for the
model.
It was also recommended that the authors include spatial autocorrelation in the model, or at the
very least test if there is autocorrelation in the residuals using a Moran’s I test. The authors
agreed to this.
When the purpose of modelling is to identify areas of potential habitat for a species, as in the
current study, a threshold is usually chosen to distinguish between areas predicted to have
individuals (predicted suitable habitat) and areas predicted to not have individuals (predicted
unsuitable habitat). One of the reviewers asked what the rationale was for selecting the
particular thresholds used in the study. The authors noted that they set the thresholds using a
statistical approach because they didn’t know where the threshold was in nature. They agreed
to add text to the Working Paper to describe this better.
One of the reviewers wondered what the rationale was for setting the neighbourhood distance to
850 m. The authors noted that nothing is known about movement of sea cucumbers among
populations. They chose 850 m because adults of the species have been documented in the
literature to move 5 m per day. If they moved this distance in the same direction every day for
4
half a year (they don’t move a lot in the winter), they would be close to 850 m. This scale may
actually be larger than biologically relevant and smaller than a management scale. There are
computational constraints as well. A smaller spatial resolution would have created too many
habitat patches and would have significantly increased model run time. The authors agreed to
clarify their choice of neighbourhood distance in the text of the Working Paper.
One of the reviewers had concerns that the GRIP function may skew representation of the
biological input parameters across the sampling space. They felt that this issue could be
addressed with coding in a sampling design based on Latin hypercube sampling. The authors
pointed out that there was no point in exploring ranges outside those requested by Management
and the Committee agreed with this. The authors agreed to add text to the Working Paper as to
why they chose the particular distributions of the biological input parameters.
One of the reviewers thought that it would be interesting to see how sensitive the model
outcomes were to different dispersal models. The Committee noted that this request was
outside the scope of the paper. Various benthic invertebrate species which could be modeled
were already mentioned in the paper and the Committee felt that this was sufficient. No changes
to the Working Paper were requested.
One reviewer found the description of the current management regime confusing with all the
various terms (i.e. fishery management area, sub-area, quota management area) and
suggested that a map would be helpful to show these areas and clarify the management
structure. The authors agreed to adding a map and clarifying the text in the Working Paper in
regards to the description of the various management areas.
It was also suggested that the clarity of the Fisheries Management section of the Working Paper
could be improved by including a decision tree that illustrates the different spatial scales of
simulated management actions. Specifically, this could help clarify at which scale (population
level or PFMA) different fisheries are varied, whether the two types of harvesting are varied
through simulation concurrently, and their spatial arrangement (i.e. does harvesting occur only
at the population level or both?). The authors agreed to add clarifying text to the Working Paper
to address this.
It was questioned why interactions were not included in the BRT analysis. The authors
commented that they did not have enough time to do this but that they could be included in
future runs of the model. The committee was fine with this justification and no changes of the
Working Paper were requested.
COMMITTEE COMMENTS AND QUESTIONS
The Committee questioned how the observed sea cucumber densities were calculated, as the
density calculation will depend on what tide level the survey transects were done at. There was
a suggestion that transects be truncated at a certain depth. The authors confirmed that there
was no depth truncation on density calculations and agreed to add clarifying text to the Working
Paper that correct density calculations should be used in future model runs.
Both the Committee and one of the reviewers questioned why data on bottom-type, sediment
facies, and/or sediment physico-chemical properties were not included in the habitat suitability
sub-model, as sea cucumbers are often affiliated with certain substratum characteristics. The
authors noted that this sort of data was not available for all locations within the area of study.
They also remarked that there was a time constraint, as extracting this data would have
required a lot of upfront processing of the data files. There was Committee consensus that a
recommendation of the SAR be to assess importance of bottom type or other benthic factors on
future model outcomes if data become available for area of analysis. The authors agreed to
5
adding clarification in the Working Paper regarding bottom-type data availability and the
reasoning for not including it in the model. They also agreed to include a new figure showing
observed presence/observed absence data with clarifying text.
Similar discussion regarding why exposure was not included in the model occurred. Again, the
authors noted that this sort of data was not available for all locations within the area of study.
There was Committee consensus that a recommendation of the SAR be to assess importance
of exposure on future model outcomes if data become available for area of analysis.
One Committee member requested that further description of how BRT analysis works be
added to the Working Paper, but there was Committee consensus that this need not be done,
given the level of detail already in the paper and the inclusion of suitable references.
The Committee requested clarification of Table 2-4 (deviances of performance statistics for all
sensitivity runs in the habitat suitability and density sub-models). It was noted that deviance
does not change that much, even with no modelled data. It was recommended by the
Committee that clarification be added on what deviance means in the text of the Working Paper.
It also recommended removal of Table 2-4 and provision of text to explain the small range of
deviance. The authors agreed to these suggested changes.
The Committee questioned why particular seasons for temperature/salinity data were chosen to
input into the habitat suitability and density sub-models and asked if there were potential
problems with intercorrelations among times or between temperature and salinity (as all
environmental data were from the same source, an oceanographic circulation model). The
authors noted that they simply used variables that were available in Dr. Mike Foreman’s
oceanographic circulation model for the area of study. There was Committee consensus that the
data do not have to be independent to be used in the models. No changes were required to the
Working Paper.
There was discussion surrounding the paradox of predicted high densities of sea cucumbers at
heads of inlets (and other areas) where poor habitat is predicted. This was a major point of two
of the reviewers as well. There was Committee consensus that a new figure be added to the
Working Paper showing the relationship between predicted and observed densities. This should
be accompanied by some clarifying text as to why predicted and observed do not always match
up. The authors agreed to these changes. Also, the Committee recommended inserting text
explaining the overlap in predicted habitat suitability and predicted density in the Proceedings.
Here is the relevant sentence from the revised Working Paper: “There is a relationship between
the observed densities of sea cucumbers and the predicted densities (Figure 2-8). However,
when observed densities are high the predicted values are lower than the observed values and
when observed densities are low the model results over-estimate density more often than not.”
In a similar vein, there was further discussion concerning the difference between occupied and
unoccupied habitat. For instance, there might be no sea cucumbers evident at the time of the
survey, but it could be suitable habitat and they may be there at a later date (or vice versa). The
Committee recommended adding text to the Working Paper about the issue and how it might
influence the model. The Committee also suggested adding a future-work recommendation in
the SAR: to examine different inputs of habitat patches on model output. The authors agreed to
these revisions.
The Committee debated at length the inclusion of results of other model runs that were not
included in the Working Paper (i.e. increased fishing effort) and text to describe the methods
and results. It was concluded that there was no point in exploring ranges outside those
requested by DFO Management and that the additional runs should not be included in the
document. It was later noted by one of the authors that the additional model runs that were
6
conducted but not included in the Working Paper were precursory runs (not full runs) and really
should not be included in the document as they are very preliminary. Full runs would require a
substantial amount of computational time and effort and would expand the length of the Working
Paper substantially if added.
There was discussion surrounding the spatial scale at which the fishery is managed and the
distribution of fishing effort and how these factors may affect the model outcome. There was
Committee consensus that a recommendation of the SAR be to evaluate if this is indeed an
important issue to be further examined in future work, in light of the patchiness of populations.
The Working Paper should clarify that fishing is implemented uniformly in the model but not in
reality (it is patchy). There should also be some clarifying text in the Working Paper explaining
the limitations of grid size and why a 300-m grid was used as opposed to a smaller scale. The
authors agreed to these suggested changes. The Committee noted that the 300-m grid was
chosen based on the scale used in Dr. Mike Foreman’s oceanographic circulation model.
One Committee member noted that he has seen declines in sea cucumber populations in areas
with very low fishing pressure that do not match model predictions. There was Committee
consensus that this be noted in the Proceedings, but not in the Working Paper.
Another Committee member questioned whether the modeling tool was robust enough to be
used in making management decisions. There was Committee consensus that it was.
There were concerns from the Committee around the many required assumptions of the data
used in the model and the justification of its use in making management decisions. There was
Committee consensus that the Working Paper should clarify text in regards to the output of the
model and how it can be used to illustrate the effects of certain variables, given the numerous
assumptions of the data and the area of study. The authors agreed to make this clarification.
The Committee agreed that the SAR should recommend that further analysis, outside the
context of the present paper, would be needed to validate the results of the model before its use
in making management decisions.
One Committee member wondered if different climate change and ocean acidification scenarios
could be included in the model. The authors noted that this could be done in future runs of the
model if the relevant data were available. No changes were required in the Working Paper.
There were questions surrounding the intent of management for FCANs for single species and
their integration with MPA Networks. There was Committee consensus, however, that this was
not in the TOR and was not the intent of the Working Paper. This would be for further
Management discussion. No changes were required in the Working Paper.
CONCLUSIONS AND RECOMMENDATIONS
The Working Paper was accepted with the following suggested revisions:
•
Need to examine publications on temperature for better references and more nuanced
information on temperature effects in temperate sea cucumber species. This information is
to be added to the text of the Working Paper.
•
Need to add more discussion surrounding the choice of BRT over other ensemble methods
such as bagging, random forest, or MAXENT or a consensus model.
•
Correlated predictor variables are common in these kinds of studies. It would be good to
know what the correlated variables were in the analysis and how the results would be
affected if one or more of these correlated variables were taken out of the analysis. Need to
add some clarifying text.
7
•
Need to add more rationale as to the choice of depth for the habitat suitability and density
sub-models and see if there is enough data to run the analysis when depth is restricted to
15 m.
•
Clarifying text is required to explain why there is an increase in percent decline for
populations greater than 45 million.
•
Should incorporate % deviance and AUC, to measure the performance of the model, and
explain the results.
•
Spatial autocorrelation should be included in the model or at the very least test if there is
autocorrelation in the residuals using a Moran’s I test.
•
A better description of how thresholds [chosen to distinguish between areas predicted to
have individuals (predicted suitable habitat) and areas predicted to not have individuals
(predicted unsuitable habitat)] were set is required.
•
The choice of 850 m as the neighbourhood distance needs to be more clearly justified.
•
The choice of the particular distributions of the biological input parameters needs to be more
clearly justified.
•
A map showing the various fishery management sub-areas is required as is text clarifying
the various levels of management areas (fishery management area, sub-area, quota
management area) and spatial scales.
•
Need to clarify that there was no depth truncation on density calculations and that correct
density calculations should be used in future model runs.
•
Should clarify bottom-type data availability and the reasoning for not including it in the
model.
•
Add new figure showing observed presence/observed absence data with clarifying text.
•
Table 2-4 (showing performance statistics for all sensitivity runs) should be removed, but
there should be added text to better explain the reason for the small range of deviance.
•
Add new figure showing the relationship between predicted and observed densities with
clarifying text.
•
There might be no sea cucumbers evident at the time of the survey, but it could be suitable
habitat and they may be there at a later date (or vice versa). Some discussion about this
issue and how it might influence the model is required.
•
Should clarify that fishing is implemented uniformly in the model but not in reality. There
should also be some clarifying text explaining the limitations of grid size and why a 300-m
grid was used as opposed to a smaller scale.
•
Need to clarify text in regards to the output of the model and how it can be used to illustrate
the effects of certain variables, given the numerous assumptions of the data and the area of
study.
In addition, the Committee recommended the following for future work to be included as
recommendations in the SAR:
•
Need to assess importance of bottom type (or other benthic factors) and exposure on future
model outcomes if data become available for area of analysis.
•
Need to examine the effect of habitat patchiness on model output.
8
•
Need for further analysis in a particular study area, outside the context of the present
Working Paper, to validate the results of the simulations tool before its use in making
management decisions.
•
The next stage of running the simulation tool fully, using a specified location and species
and having the results reviewed by CSAS to review the application and types of advice
stemming from these tools, was recommended.
ACKNOWLEDGEMENTS
Matthew Slater (Department for Marine Aquaculture, imare - Institute for Marine Resources,
Bremerhaven, Germany), Stephen Smith (Bedford Institute of Oceanography, Fisheries and
Oceans Canada, Halifax, Nova Scotia), and Ilona Naujokaitis-Lewis (Department of Ecology
and Evolutionary Biology, University of Toronto, Toronto, Ontario) each provided a thorough
written review of the Working Paper. Their efforts in providing this feedback to the committee
and authors are greatly appreciated. Also, the Committee thanks Dan Curtis for acting as
rapporteur for the meeting.
REFERENCES
DFO. 2011. Assessment framework and management advice for the British Columbia giant red
sea cucumber (Parastichopus californicus) fishery. DFO Can. Sci. Advis. Sec. Sci. Advis.
Rep. 2010/080.
DFO. 2012. Pacific Region Integrated Fisheries Management Plan Sea Cucumber by Dive
October 1, 2012 to September 30, 2013.
Duprey, N., Hand, C., Lochead, J., and Hajas, W. 2011. Assessment Framework for Sea
Cucumber (Parastichopus californicus) in British Columbia. DFO Can. Sci. Advis. Sec. Res.
Doc. 2010/105. vi + 38 p.
9
APPENDIX A: AGENDA
Canadian Science Advisory Secretariat
Centre for Science Advice Pacific
Regional Peer Review (RPR) Meeting
Simulation Modelling Tools to Evaluate Alternative Fishery Closure Network Designs for
Shallow-water Benthic Invertebrates in British Columbia
October 23-24, 2013
Pacific Biological Station, Nanaimo, British Columbia
Chairperson: Chris Pearce
Wednesday, October 23
Time
Subject
Presenter
0900
Introductions
• Review Agenda & Housekeeping
• CSAS Overview and Procedures
• Review Terms of Reference
Chair
0945
Presentation of Working Paper
Authors
1030
Break
1045
First Review
Reviewer #1
1115
Second Review
Reviewer #2
1200
Lunch
1300
Third Review
Reviewer #3
1345
General Questions
Participants
1430
Break
1445
Discussion & Review of Working Paper
1600
Adjourn
Participants
Thursday, October 24
Time
Subject
Presenter
0900
Welcome & Introductions
Chair
0915
Recap of Day 1
Participants
0945
Discussion & Review of Working Paper
Participants
10
Time
Subject
Presenter
1030
Break
1050
Discussion & Review of Working Paper
1200
Lunch
1300
Forming Key Conclusions and Advice for Science
Advisory Report
1430
Break
1445
Finalizing Science Advisory Report
1600
Adjourn meeting
11
Participants
Participants
Participants
APPENDIX B: PARTICIPANTS
Last Name
First Name
Affiliation
James
Laura
Dominique
Jon
Dan
Janelle
Lyanne
Sarah
Anya
Nicholas
Jessica
Ken
Graham
Wayne
Claudia
Gabrielle
Kate
Joanne
Dan
Janet
Miriam
Chris
Ian
Pauline
Juanita
Paul
Stephen
Erin
Zane
DFO, Science
DFO, Science
DFO, Science
DFO, FAM
DFO, Science
DFO, Science
DFO, Science
DFO, Science
DFO, Science
DFO, Science
DFO, Science
DFO, Science
DFO, Science
DFO, Science
DFO, Science
DFO, FAM
DFO, Science
DFO, Science
DFO, Science
DFO, Science
DFO, Science
DFO, Science
DFO, Science
DFO, FAM
DFO, FAM
DFO, FAM
DFO, Science
DFO, FAM
DFO, Science
Doug
Karin
Dennis
Ken
Grant
Ken
Matthew
Province of BC
Living Oceans Society
Province of BC
Kitasoo/Xaixais First Nation
Underwater Harvesters Association
Pacific Sea Cucumber Harvesters Association
imare - Institute for Marine Resources
DFO
Boutillier
Brown
Bureau
Chamberlain
Curtis
Curtis
Curtis
Davies
Dunham
Duprey
Finney
Fong
Gillespie
Hajas
Hand
Kosmider
Ladell
Lessard
Leus
Lochead
O
Pearce
Perry
Ridings
Rogers
Ryall
Smith*
Wylie
Zhang
External
Biffard
Bodtker
Chalmers
Cripps
Dovey
Ridgway
Slater*
* Provided written reviews on the Working Paper
12
APPENDIX C: TERMS OF REFERENCE
A Simulation Modelling Tool to Evaluate Alternative Fishery Closure Network
Designs for Shallow-water Benthic Invertebrates in British Columbia
Regional Peer Review - Pacific Region
October 23-24, 2013
Nanaimo, British Columbia
Chairperson: Chris Pearce
Context
The use of fishery closures is a component of the management strategy for the commercial sea
cucumber (Parastichopus californicus) fishery (DFO 2011, DFO 2012). Of the many benefits
that may be derived from reserves, the primary purpose for existing fishery closures is to protect
a portion of the P. californicus population as a safeguard against overfishing, given uncertainties
in the current understanding of biology and population dynamics.
In British Columbia, a total of 13 areas where commercial sea cucumber fishing is prohibited
have been in place since 2008 (DFO 2012). To date, the number, size and location of these
fishery closures have been determined using a set of arbitrary criteria developed from in-house
knowledge and expertise, and includes representativeness of surrounding harvested areas,
ease of definition for management purposes, size appropriate for monitoring and an arbitrary
target percent of harvestable shoreline of 20% (Duprey et al. 2011). Science advice was
requested by Fisheries Management to provide a scientifically-sound and transparent process
for the development of a coastwide network of fishery closures for shallow-water benthic
invertebrates.
The primary goal of this work is to develop a simulation-modelling approach to evaluate
alternative fishery closure network designs and fishery management scenarios for
commercially-harvested low-mobility benthic invertebrates, with a particular focus on P.
californicus, using specified performance indicators. Simulation results will be evaluated for their
potential to provide advice on size, spacing and configuration of fishery closures in areas with
limited data (e.g. on species distribution, habitat suitability and larval dispersal patterns). It is
intended that the simulation tool will be flexible enough to allow its application to other fisheries
for low- or no-mobility benthic invertebrate species for which fishery closures or other
management measures may be defined. The specific outputs of this work will also inform
current work that is underway to establish a network of MPAs in British Columbia, as per the
draft Canada-BC MPA Network Strategy and ongoing marine spatial planning processes, such
as the Pacific North Coast Integrated Management Area (PNCIMA).
Objectives
This Regional Peer Review Meeting (RPR) will review and provide advice based on the
following working paper:
Duprey, N.M.T., Finney, J., Curtis, J., Hand, C.M. Simulation modelling tools to evaluate
alternative fishery closure network designs for shallow-water benthic invertebrates in British
Columbia. CSAP Working Paper 2013/P63.
The objectives of this Regional Peer Review Meeting (RPR) are to:
1. Assess the performance of the simulation tool’s ability to evaluate fishery closure network
designs for sea cucumbers that vary in number, size and location, as well as under various
scenarios of data availability;
13
2. Assess the performance of the simulation tool’s ability to evaluate alternative fishery closure
network designs for sea cucumbers based on a range of performance measures and
plausible commercial and First Nations fishery management scenarios.
3. Evaluate uncertainty in parameter assumptions and simulation tool results and, based on
sensitivity analysis, provide recommendations for ways to reduce uncertainty;
4. Assess the applicability of the simulation tool to other low-mobility, shallow-water benthic
invertebrates.
5. Provide a discussion on the suitability of proxies or alternative methods to identify candidate
commercial fishery closure locations for low-mobility benthic invertebrates; and,
6. Provide recommendations for research and monitoring of biological trends to evaluate their
effectiveness in achieving conservation and fishery management objectives.
Expected publications
•
•
•
CSAS Science Advisory Report (1)
CSAS Research Document (1)
CSAS Proceedings (1)
Participation
•
•
•
•
•
•
•
•
•
DFO Science
DFO Fisheries Management
DFO Oceans
DFO Habitat
Province of BC
Commercial and recreational fishing interests
First Nations
Non-government organizations
Academia
References Cited
DFO. 2011. Assessment framework and management advice for the British Columbia giant red
sea cucumber (Parastichopus californicus) fishery. DFO Can. Sci. Advis. Sec. Sci. Advis.
Rep. 2010/080.
DFO. 2012. Pacific Region Integrated Fisheries Management Plan Sea Cucumber by Dive
October 1, 2012 to September 30, 2013.
Duprey, N., Hand, C., Lochead, J., and Hajas, W. 2011. Assessment Framework for Sea
Cucumber (Parastichopus californicus) in British Columbia. DFO Can. Sci. Advis. Sec. Res.
Doc. 2010/105. vi + 38 p.
14
APPENDIX D: ABSTRACT FROM WORKING PAPER
The suitability of alternative tools or proxies for designing fisheries closure area networks
depends first and foremost on the management objectives and how these are articulated in an
operational sense. For instance map-based approaches (e.g. habitat suitability models,
MARXAN analyses) may be appropriate in cases when management objectives are expressed
simply in terms of a target area to be incorporated into the network. Alternatively, more
sophisticated simulation models that account for dynamic interactions between fisheries,
populations, and environmental processes (e.g. ocean circulation) may be required when
management objectives are expressed in terms of changes in the patterns of distribution or
abundance of species. In this paper we describe and apply a set of quantitative tools that can
be used to provide advice on the number, size and spacing of fishery closure areas. The tools
build on software packages and programs (ArcGIS, RAMAS, R, GRIP) that are currently
available and being applied to address a broad range of marine spatial planning objectives.
Together these tools form key components of a simulator that couples four sub-models of
habitat suitability, metapopulation dynamics, dispersal and fisheries management. We apply
these tools to demonstrate how they might be used to inform decisions on fishery closure area
design in Area 12 using Parastichopus californicus as a case study species. We show how the
results of a model sensitivity analysis can be used to evaluate alternative network designs using
a range of performance criteria. We conclude with a discussion of how the set of tools may be
applied to address a broad range of spatial management questions for a broad range of species
in any area provided sufficient data are available.
15
APPENDIX E: WRITTEN REVIEWS
REVIEWER 1
M.J. Slater – Head of Department for Marine Aquaculture, imare Institute for Marine
Resources, Bremerhaven, Germany
Date: 16 October 2013
CSAS Working Paper: 2013/P63
Working Paper Title: Simulation Modelling Tools to Evaluate Alternative Fishery Closure Area
Network Designs for Shallow-water Benthic Invertebrates in British Columbia
Summary:
This paper addresses the challenging and timely topic of marine reserve (no-take areas/fishery
closure areas/marine protected areas) specifically designated for the protection or management
of stocks of one low-mobility benthic invertebrate. Given the historical “boom and bust” nature of
fisheries for high value benthic species, especially sea cucumber fisheries, rapid implementation
of effective fishery measures is required as soon as possible after exploitation commences.
With this in mind, this study is likely to find practical application within fishery management and
is, at the same time, inherently challenged by a lack of available biological knowledge about the
species and populations being modelled. Overall the authors approach the modelling task and
in particular the gathering of basic biological knowledge for the case study Parastichopus
californicus thoroughly given the large amount of reference literature available and manage the
significant challenge of making and defending assumptions in their model(s) in a pragmatic and
convincing manner. They use ground-truthed density data to refine their models and apply their
model to a valid fishery area, although I cannot judge its overall representativeness within the
managed area overall. I have some comments or concerns about some of the assumptions and
am in agreement with the authors regarding specific data types which are sorely lacking in the
development of the habitat and density models, both of which I will address in detail below. On a
less positive note, in certain parts of the text the authors attempt to address apparent
contradictions in their models however fail to offer viable reasons why contradictions occur. In
addition, while the paper ultimately identifies key elements in Fishery Closure Area Networks
design which may improve network efficacy for the candidate species, it would have benefitted
from the testing and comparison of example networks, which vary spatially or temporally, with
visual representation of network types and predicted performance. This may however be an
unfounded criticism as my knowledge of the modelling process is limited and the examples may
have required significant time inputs. The authors are able to list the effect, however, of
management variables (p78-80), so it is surprising that no “optimal scenario” could be
calculated. On the other hand, the paper benefits from the excellent inclusion of existing
management measures in the sensitivity analysis (p53). Overall the paper addresses and fulfils
the clearly stated aims and objectives and in doing so highlights, and partially tackles, the many
challenges managers and modellers face in rapidly addressing invertebrate fishery
management requirements with minimal viable biological and/ or physical data for the species
and areas in question.
Specific review questions:
•
Is the purpose of the working paper clearly stated?
Yes
•
Are the data and methods adequate to support the conclusions?
16
The methods are in my opinion suitable however the data applied are non-existent in many
case, and as the authors note, some data sets have been excluded which are assumed to be
extremely important to understanding the density and habitat distribution of the species. This
applies to bottom type / facies in particular.
•
Are the data and methods explained in sufficient detail to properly evaluate the conclusions?
Yes
•
If the document presents advice to decision-makers, are the recommendations provided in a
useable form, and does the advice reflect the uncertainty in the data, analysis or process?
The final discussion could be presented in a more accessible manner to managers who may
benefit from a comparison of the modelled impact of specific, existing or proposed network
types.
•
Can you suggest additional areas of research that are needed to improve our assessment
abilities?
The authors final sentence notes the “drawback of the simulation approach is the level of
expertise required and time required to carry out the analysis”. I would add to this the drawback
of the huge amount of assumptions needed to be made in the absence of viable biological data
for the species and physical data for the study area, let alone the area to which the fishery
management measures apply. While this is an all too common problem, it requires addressing,
particularly with regard to the fundamental biological data available for this commercially and
culturally (from the perspective of First Nation traditional fishers) important, heavily exploited
sea cucumber species.
Specific comments on the paper and requested changes:
References are partially incomplete throughout.
p7/8 please clarify the use of the term marine reserve and clearly introduce the term Fishery
Closure Area Network as a reserve-type fishery management measure in the first paragraphs.
The terminology for reserves is now diverse internationally and can be confusing.
p7: end of paragraph one, insurance policy aspect is incompletely argued and referenced.
P12: The environmental data are incomplete without indications of bottom type, sediment facies
and several sediment physic-chemical characteristics or deposition characteristics (although this
species obviously differs from many sea cucumber species in its preference for hard
substrates).
P13: The use of the word “handle” is colloquial.
P14: Work by Zamora and Eeckhaut also support assumption that salinity variations are
detrimental to A. mollis and H. scabra respectively.
P14: I disagree with the assumptions of importance of temperature in your model given the
small temperature ranges provided in Table 2-1. The cited literature is primarily related to
temperature extremes well beyond the normal range for the species investigated. It may be
worth investigating the growth and behavioural data in Zamora’s publications on temperature for
better references and more nuanced information on temperature effects in temperate species.
P20: Final paragraph is poorly structured and explained.
P22: Section 2.3.4.3 delete first sentence and move figure ref to sentence two.
17
P23: Would your density model benefit from ground-truthing predicted densities in non-surveyed
areas post-modelling.
P26: The contradiction that salinity is unimportant in your model is not addressed in the
explanation of habitat suitability and seasonal fluctuations at inlet heads.
P27/28: Here key contradictions and limitations are recognized and at least partially addressed.
The recommendations for application in the case of the interplay between the models is good. I
believe the depth data is invalid beyond the survey depth however this limitation is clearly
recognized in the text.
P39: If the animals in j2 are larger that 3-4 cm then I would expect an increase in survival over
j1 - a more accurate figure for g23 may be between 0.3 and 0.4.
P42: fecundity seems extremely high, some spawning and hatchery attempts/successes have
been made with this species, are there no egg counts available?
P43: In natural spawning events I would expect less than 5% of released eggs to be non-viable
on release.
P50: I agree that modelling current influence would be both extremely time-consuming and
potentially counterproductive at the scales applied.
REVIEWER 2
Stephen J. Smith, DFO, Bedford Institute of Oceanography
Date: October 16, 2013
CSAS Working Paper: 2013/P63
Working Paper Title: Simulation Modelling Tools to Evaluate Alternative Fishery Closure Area
Network Designs for Shallow-water Benthic Invertebrates in British Columbia
Right up front I have to say that I am very impressed with this report. While the authors note in a
number of places in the document that they were unable to complete various aspects of the
analysis or modelling, they should still see their work as a tangible contribution to how one
should approach the problem of evaluating spatial management measures for benthic
populations. Limiting attention to single species Fishery Closure Area Networks rather than
communities of species makes the work more manageable in some way but no less in terms of
the work involved in putting together the information on the physical variables, life history
parameters, developing the population models and methods for evaluating the management
methods.
I mainly concentrate my review on the habitat suitability analysis because this underlies the rest
of the work and it is the one area in this paper that I have most expertise in.
Habitat Suitability maps
Recently, machine-learning methods have become popular for developing habitat suitability
models relating large numbers of predictive variables with spatial information on species
presence, presence-absence and densities. The advantage of using these methods is that they
can develop quite complex relationships between the predictors and response in an
unsupervised fashion. However, these methods are also seen as `black-boxes’ with their
performance depending upon how various method-specific “parameters” are tuned for each
application; e.g., the tuning of parameters nt, bf, lr and tc for the boosted regression trees used
in this study.
18
The parameter tc needs to some more explanation here. This report defines this measure of
tree complexity as the number of nodes or splits in the tree and as a control on including
interactions. Elith et al. (2008) define “… the tree complexity (tc) controls whether interactions
are fitted: a tc of 1 (single decision stump; two terminal nodes) fits an additive model, a tc of two
fits a model with up to two-way interactions, and so on.” In the classic single classification or
regression tree, each split is binary and interactions are possible as soon the root node has
been defined. That is, interactions occur when branches from the same node have different
splitting predictors further down the tree. Therefore a tc=1 would only result in one predictor
being chosen for each tree, hardly an additive model in the true sense. The reader could benefit
from a graphical representation of what a tc=3 (or 7) tree would look like for the
presence/absence and density applications. How strongly correlated is the bag fraction with the
tc parameter? It is likely that the number of interaction terms included is a function of the
variability induced by subsampling the data.
This complexity in application results in difficulty in explaining these kinds of methods, in
anticipating the results and in evaluating the method’s performance. The reader could benefit
from more discussion about why boosting was chosen over other ensemble methods such as
bagging, random forest or MAXENT. All of the method comparisons that I have seen involve
performance evaluations on selected real data sets or on artificial data sets. As Elith et al.
(2006) point out model performance can be substantially affected by the manner in which a
particular method is implemented and the results of comparative studies are limited by how
each method is set up and by the features of the data sets used. A number of studies
comparing bagging, boosting and random forest all variants of ensemble methods based on
regression or classification trees report mixed, equivocal or minor differences between them
(Boinee et al. 2006, Ogutu et al. 2010, Shataee et al. 2011, Ghimire et al. 2012). So what drew
you to boosting and why would you recommend this method to others? Compared to boosted
regression trees, random forest only requires two parameters (number of trees and number of
predictors in sample) and in my experience is an easier method to explain.
Correlated predictor variables are common in these kinds of studies. We would expect some
combinations of correlated variables, e.g., temperatures in different seasons, depth and current
speed, depth and temperature. Advice in the literature varies from leaving all of the variables in
to taking all correlated variables out. The method used in this study of apparently removing each
of the correlated predictors one at a time (Table 2-4) is difficult to evaluate because we do not
know which variables they were and whether they were originally judged to be important (as in
Figure 2-2). We also don’t know if their removal changed the contribution level of other predictor
variables in the model. Strobl et al. (2008) demonstrated the impact of correlated predictors on
random forest models where correlated predictors will have inflated importance relative to
uncorrelated predictors (see Ellis et al. (2012) for a discussion of conditional measures to deal
with correlation for the gradient forest method).
I would have liked to have seen a plot of predicted habitat suitability versus predicted density to
further evaluate the results of this model. Were the higher densities associated with the higher
habitat suitability’s? Could the authors speculate on other variables not measured here that
might explain why the predicted densities consistently underestimated the observed densities?
I am confused by what depths you actually had data from. The depth range for predictor
variables was given as 1–650 m, but the surveys for the sea cucumber data seem to be limited
to 15 m. However, the relationship between depth and the two kinds of responses in the models
were identified as plateauing at 25 m for habitat suitability and 50 m for density. Sensitivity to
depth was tested by limiting the analysis to areas shallower than 30 m depth. We are then told
that the results should be interpreted with caution because sea cucumber data was really only
available up to around 15 m. So am I correct in assuming that there was no sea cucumber data
19
deeper than 15 m even though the predictor variables were available for much deeper depths?
If so, how do you come up with plateauing relationships at depths deeper than you have
response data for?
Simulation tool & Metapopulation dynamics
I do not have any experience with the software packages used here (e.g., RAMAS GIS
software) and cannot really comment on the appropriateness of their application. I have built
habitat-based population dynamic models but not with the complex metapopulation dynamics as
was done here. It was very difficult to boil down all of the structure and parameter combinations
down to a simple evaluation of productivity > exploitation, so all is well scenario. I assume that
something like this comparison underlay the establishment of the annual rate of exploitation of
4.2% (or 6.7%) versus 10% over three years but these rates are applied to the lower 90%
bound on the population estimate and so the actual rates may be quite small. How does the
0.66 correction used to convert the mean to the 90% lower bound estimate plus the recreational
and aboriginal exploitation rates jive with the 0.18% rate of increase used for the baseline
model?
With respect to differences between maps used by fisheries management and science — I hear
you.We learned the hard way that positions in Regulations (Variation orders, etc.) are still in
NAD 27.
Fisheries Management model
I cannot say that I understand what the results of the boosted tree regression of the simulation
results are telling me with respect to the relationship between initial abundance and percent
decline. Why the increase in percent decline for populations greater than 45 million? All
populations were assumed to be at carrying capacity initially, so what is so special about 45
million — mean to variance relationship?
References
Boinee, P., De Angelis, A., and Foresti, G.L. 2006. Meta random forests. International Journal of
Information and Mathematical Sciences 2: 138–147.
Elith, J., Graham, C. H., Anderson, R. P., Dudı´k, M., Ferrier, S., Guisan, A., Hijmans, R. J.,
Huettmann, F., Leathwick, J. R., Lehmann, A., Li, J., Lohmann, L. G., Loiselle, B. A.,
Manion, G., Moritz, C., Nakamura, M., Nakazawa, Y., Overton, J. McC., Peterson, A. T.,
Phillips, S. J., Richardson, K. S., Scachetti-Pereira, R., Schapire, R. E., Sobero´n, J.,
Williams, S., Wisz, M. S.,and Zimmermann, N. E. 2006. Novel methods improve prediction
of spatial data. Ecography 29:129–151.
Elith, J., Leathwick, J.R., and Hastie, T. 2008. A working guide to boosted regression trees.
Journal of Animal Ecology 77:802–813.
Ellis, N., S.J. Smith, and C.R. Pitcher. 2012. Gradient forests: calculating importance gradients
on physical predictors. Ecology. 93: 156–168.
Ghimire, B., Rogan, J., Galiano, V. R., Panday, P., and Neeti, N. 2012. An evaluation of
bagging, boosting, and random forests for land-cover classification in Cape Cod,
Massachusetts, USA. GIScience & Remote Sensing. 49: 623–643.
Ogutu, J. O., Piepho, H.-P., Schulz-Streeck, T. 2010. A comparison of random forests, boosting
and support vector machines for genomic selection. BMC Proceedings 2011, 5(Suppl
3):S11. (Accessed December 15, 2015)
20
Shataee, S., Weinaker, H., and Babanejad, M. 2011. Plot-level forest volume estimation using
airborne laser scanner and tm data, comparison of boosting and random forest tree
regression algorithms. Procedia Environmental Sciences 7: 68–73.
Strobl, C., A. Boulesteix, T. Kneib, T. Augustin, and A. Zeileis. 2008. Conditional variable
importance for random forests. BMC Bioinformatics 9:307–317.
REVIEWER 3
Ilona Naujokaitis-Lewis, University of Toronto
Date: October 17, 2013
CSAS Working Paper: 2013/P63
Working Paper Title: Simulation Modelling Tools to Evaluate Alternative Fishery Closure Area
Network Designs for Shallow-water Benthic Invertebrates in British Columbia
Review of Simulation Methods and Outcomes
I have divided my review into the four main sections reflected in the simulation modelling
component of the report. I simultaneously provide comments on the methods and results,
however, the comments are more focused on the methods themselves. Overall, I found this
work really interesting, comprehensive, and methodologically relevant to evaluating alternative
closure network designs. In addition to the general comments below, I have included edits and
further questions (or minor relevance) in the text of the report.
Habitat Suitability Model
To start, this section should be renamed as it is not focused on the habitat suitability model itself
but rather on parameterizing the RAMAS Spatial Data submodule, which uses the suitability
model’s predictive map of habitat suitability to derive the spatial structure of the metapopulation.
My major concern with this section is whether the BRT model adequately models habitat
suitability for the focal species, given the uncertainties that can be present in species distribution
model methods. It is well acknowledged in the literature that there can be large variability
associated with different modelling methods. One way to address this is to develop a consensus
model based on a combination of a range of distribution model methods. In many cases, the
resulting model has much higher predictive accuracy. Given that there were some concerns with
the predictions conforming to expert knowledge and survey outcomes of the distribution of the
species, using a suite of distribution modelling methods and developing a consensus model
might address this issue. Furthermore, this might be a way to extend the sensitivity analysis to
the distribution/suitability model, which currently only consists of applying different thresholds to
one model’s predictions. One reference to consult is Araújo MB, New M (2007) Ensemble
forecasting of species distributions. Trends in ecology & evolution, 22, 42–47.
As well, I would include a few additional performance metrics for model assessment.
Specifically, the % deviance explained by the model is a measure of the goodness of fit
between the observed and modelled values. Secondly, the area under the receiver operating
curve (AUC), is a threshold independent metric that combines the trade-off between sensitivity
(the true positive proportion) against the false positive proportion. Another metric that could be
considered is the True Skill Statistic, which accounts for both omission and commission errors,
and ranges from −1 to +1, where +1 indicates perfect agreement and 0 random fit. See Allouche
et al 2006. Assessing the accuracy of species distribution models : prevalence, kappa and the
true skill statistic (TSS). Journal of Applied Ecology, 46, 1223–1232.
21
And finally, I would recommend including spatial autocorrelation in the model, or at the very
least testing if there is autocorrelation in the residuals using a Moran’s I test.
A few outstanding questions I had about this section (and section 1 on model development) that
might influence the performance of the models:
1. Was the density data converted to detection/non-detection data, meaning that data were the
same over space and time, just one was simplified? Were the data synthesized (averaged?)
across the three survey years? Did the predictor variables reflect that 3-year period , as best
as possible?
2. Are there any expected relationships between the different predictors and each of the
response variables? And how might you expect these to be different as a function of the
response, ie factors influencing presence/absence vs abundance data are likely to differ.
Would you expect there to be a relationship between predictions of the 2 models, is
predicted habitat suitability (i.e. probability of occurrence) positively related to predicted
habitat quality (i.e. density)? This is an important point as outcomes of each of these models
is used to inform the metapopulation dynamics model either as the spatial layer used to
define the metapopulation structure, and as the estimates of initial abundances of the
demographic model.
3. What is the rational for selecting the thresholds? I would think about tradeoffs associated
with thresholds that favor error of commissions (TYPE I) vs error of omission (Type II),
keeping in mind the objective of the habitat model/metapopulation dynamics model.
Table 3-1 has a column ‘initial abundances’. The table is cited in text in the section on the
Habitat suitability of the simulation section, but there is no mention of where these abundances
come from. I am assuming the density BRT model, but this needs to be clarified.
Metapopulation Dynamics Model
I thought this section was comprehensive as it contained all relevant information, and described
the methods clearly. I found that there was a fair bit of repetition in the text, and have
highlighted those areas in the document. The simulation methods applied are logical, and
importantly, address the sources of uncertainty. As I am not a sea cucumber biologist, it is
outside of my area of expertise to comment on the quantities of the estimated parameters,
although the methods for estimation appeared sound.
I kept on thinking if sea cucumbers conform to assumptions of metapopulation assumptions?
On a related note, on page 33 the neighbourhood distance was set to 850m: what is the rational
for this? I recognize that RAMAS has a 500 maximum number of populations, but what is the
biological basis for this decision? This can be interpreted as the spatial scale at which a
population is considered panmictic. It could represent foraging distance. The maximum
dispersal distance was specified as 30 km. Perhaps a sea cucumber biologist could comment
on this. This does bring up a large question in ecology: what is a (sub-) population or a patch?
Figure 3-5 Distribution of biological input parameters used in the sensitivity analysis. This figure
clearly illustrates the information contained in Table 3-6, Model parameters that may be varied
in a sensitivity analysis. From my experience, the way that GRIP is currently set up, it is
possible to end up with a skewed representation across the sampling space. Although this is
related in part to the number of simulations undertaken, it is possible to have some parameter
values underrepresented particularly those at the extremes of the probability distributions. It
would be really excellent to address this issue with coding in a sampling design based on latin
hypercube sampling. Perhaps this is under consideration? Instead of Figure 3.5, it might be
more informative to have some plots that will allow you to estimate the frequency distributions of
22
the more extreme parameter combinations, or apply another assessment to examine how well
the distribution space has been sampled.
Dispersal Model
This section requires explanation of how the dispersal estimates are integrated into the
metapopulation model.
It would be really interesting to see how sensitive the model outcomes are to different dispersal
models. I recognize that this is not the explicit aim here, but should dispersal based on
biophysical models of currents be available, it will provide an interesting contrast.
Maximum dispersal estimates of the focal species were derived from other species. What were
these species, and were they related species? Are the use of mean dispersal estimates
appropriate for parameterizing the maximum dispersal distance?
Fisheries Management
I found the section describing the Fisheries Management model and how parameters were
varied difficult to follow. In part this was due to many somewhat similar acronyms, and lack of
description of a few. For example, QMA and TACs were not explained in the text, and while
QMA was described for the first time in Table 3-5, it was not immediately clear how the different
areas related to one another in particular from a spatial/hierarchical level. It was not clear what
the current spatial distribution of FCAN closures is (or even if there is currently a FCAN in
place), nor where commercial closures were simulated. From page 49, “In the baseline model,
the number and location of populations included in the FCAN matches the current management
structure”. A map would definitely provide clarity on what this current management structure
translates into spatially. Given this, it was difficult for me to comment on the specifics of the
simulation methods for this section as I was unclear of the spatial distribution of the different
management areas both currently and under the simulated futures.
Another suggestion on how to improve the clarity of this section would be to include a decision
tree (or trees or some other type of figure, Figure 1) that illustrates the different spatial scales of
simulated management actions. This might illustrate in part the section on page 50 and the
parameters in Table 3-6 that I have highlighted in blue. Specifically, this could help clarify at
which scale (population level or PFMA) different fisheries are varied, whether the 2 types of
harvesting are varied through simulation concurrently and their spatial arrangement (i.e. does
harvesting occur only at the population level, or both?) . This, in addition to a map that at the
least generally defined the different scales of potential closures (QMA, PFMA, PFM subarea)
would have really helped me to understand how factors were being varied and how this related
to space. A map to complement the section describing the methods applied in the fisheries
management model would help to clarify spatial distributions of current and future potential
closures examined via simulation.
A few examples of questions that I still have post-review:
1. Were the 2 different types of harvesting (commercial vs other) implemented concurrently in
the model?
2. How was the harvest frequency and intensity varied over time and space?
3. Is ‘other fishery’ harvesting impacting all populations (or PFMs) or only those not closed to
commercial harvest?
23
Figure 1. Decision tree of alternative (spatial) fisheries management actions varied in the sensitivity
analysis
The section on FCAN metrics in the methods section (pg 55): It might be just because of
wording, but to me, these metrics are not what you are varying in the SA, but rather are metrics
derived post-SA and are a function of the resultant draws from the random distributions. If this is
the case, then this paragraph needs re-phrasing to illustrate this point and I would omit them
(propshore, area_sh, area_total) from figure 3-6.
Boosted Regression Tree
Why were interactions not included if they are important?
Other Sections
What is the scale of the fitted function in Figure 3-9?
Editorial comments
I have provided editorial comments throughout the document, but here I highlight a few major
suggestions.
1. It would have been really helpful in reviewing a document of such long length if there was a
complete table of contents in (sub-)headings visible in the navigation pane. There was only
a partial one formatted that went to the end of the Habitat Suitability Model Section. Possibly
some misformatting occurred with the different WORD versions? On the last note, for some
unknown reason there were automatic changes to the formatting of the headers that I did
not perform, so disregard those, I can’t seem to get rid of them.
2. Consider moving the section on GRIP 1 modifications to a later section as it currently
distracts from the flow and not all listed points are modifications – some are specifications.
3. The report was comprehensive but there was a fair amount of repetition and there were
many sentences that were awkward, and a few sections that were not clearly written. I have
added comments in text throughout.
24