Deliverable D1.3: NEMO Tangent & Adjoint Models (NemoTam) Reference Manual & User’s Guide Arthur Vidard, Franck Vigilant INRIA/LJK, Grenoble Charles Deltel, Rachid Benshila LOCEAN-IPSL, Paris VODA: ANR-08-COSI-016 VODA: ANR-08-COSI-016 2 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide Contents 1 2 N EMOTAM R EFERENCE M ANUAL 1.1 Introduction . . . . . . . . . . . . . . . . . . . . 1.1.1 Abstract . . . . . . . . . . . . . . . . . . 1.1.2 Introduction . . . . . . . . . . . . . . . . 1.2 Tangent and adjoint models (TAM) basics . . . . 1.2.1 TAM, but what for ? . . . . . . . . . . . 1.2.2 Tangent and Adjoint coding principle . . 1.2.3 Potential issues . . . . . . . . . . . . . . 1.3 Direct model changes . . . . . . . . . . . . . . . 1.3.1 Storing the non-linear trajectory . . . . . 1.3.2 Modified routines in the direct code . . . 1.3.3 Modified declarations in the direct code . 1.4 Tangent linear model approximations . . . . . . . 1.4.1 Active variables degraded to passive . . . 1.4.2 Simplifications . . . . . . . . . . . . . . 1.5 Hand coding status . . . . . . . . . . . . . . . . 1.6 Available options in NEMOTAM . . . . . . . . . 1.7 The generic validation interface . . . . . . . . . 1.7.1 Introduction . . . . . . . . . . . . . . . . 1.7.2 Adjoint test . . . . . . . . . . . . . . . . 1.7.3 Tangent-Linear validation . . . . . . . . 1.7.4 Tangent-Linear Hypothesis (HLT) validity 1.8 MPI parallelization . . . . . . . . . . . . . . . . 1.8.1 General features . . . . . . . . . . . . . 1.8.2 Adjoint of MPI ALLGATHER . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . N EMOTAM U SER ’ S G UIDE 2.1 Source code structuration . . . . . . . . . . . . . . . . . . . . . . 2.2 Getting started tutorial . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Installing the code using the NEMOVAR GIT repository . 2.2.2 Installing the code using the NEMOTAM SVN repository 2.2.3 Which executable for which test? . . . . . . . . . . . . . 2.3 Demonstrators . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 5 5 5 7 7 7 8 8 8 10 10 10 10 11 11 11 16 16 16 17 19 20 20 21 . . . . . . 23 23 24 24 25 26 26 VODA: ANR-08-COSI-016 2.4 2.5 4 2.3.1 The ORCA2 global configuration . . . . . . . . . . . . . . . . . . . . . 2.3.2 The GYRE regional configuration . . . . . . . . . . . . . . . . . . . . . 2.3.3 The POMME configuration (toy model for handling open boundaries) . . How-to update NEMOTAM with your contribution? . . . . . . . . . . . . . . . . 2.4.1 Coding rules for TAM . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.2 Workflow for introducing additional developments . . . . . . . . . . . . Frequently asked questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5.1 I can’t use GDB after compiling with FCM . . . . . . . . . . . . . . . . 2.5.2 I have many commas in the namelist generated by piano . . . . . . . . . 2.5.3 I have negative values in my cost function terms . . . . . . . . . . . . . 2.5.4 My run stops with ’PROBLEM WITH THE MATRIX OR WITH THE PRECONDITIONER’ on stderr . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 32 33 34 34 35 36 36 36 36 . 36 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide Chapter 1 N EMOTAM R EFERENCE M ANUAL 1.1 Introduction This document still contains empty parts. This is done on purpose since it will be the basis for the next deliverable (D1.3 NEMOTAM Manual). We will fill them in as we proceed. 1.1.1 Abstract The development of the tangent linear and adjoint models (TAM in the following) of the dynamical core of the NEMO ocean engine (NEMOTAM) is a key objective of the VODA project. TAM are widely used for variational assimilation applications, but they are also powerful tools for the analysis of physical processes, since they can be used for sensitivity analysis, parameter identification and for the computation of characteristic vectors (singular vectors, Liapunov vectors, etc.). In the framework of VODA, a work package has been set-up in order to develop a comprehensive NEMOTAM package, and to define an effective long-term development strategy for ensuring synchronisation of NEMOTAM with future NEMO releases. This is a heavy task, but it is worth the effort since NEMOTAM will benefit all NEMO users for the wide range of applications described above. Ideally, this strategy should be defined to allow NEMOTAM to adapt to future NEMO developments as quickly and as efficiently as possible, so that new releases of NEMOTAM can be made soon after new releases of NEMO. This will require careful coordination between the main development teams of NEMO, NEMOTAM and possibly NEMOVAR (INRIA, NEMO Team, CERFACS, ECMWF). 1.1.2 Introduction The NEMO ocean engine (?) was previously known as the OPA model (?). It used to have a TAM (called OPATAM), fully hand-coded and maintained mainly by A. Weaver. OPATAM was initially developed for a Pacific ocean configuration, and targeted at variational data assimilation applications in the framework of OPAVAR (??). OPATAM/OPAVAR were extended to other regional basins (Mediterranean sea (?), North Atlantic 1/3◦ (?), South Atlantic 1◦ ), to the global 5 VODA: ANR-08-COSI-016 ocean (ORCA 2◦ (?)), and were used for methodological studies such as control of the 3D model error (?), control of the surface forcing and open boundary conditions (??). OPATAM was also used for sensitivity studies (?), singular vectors (??), etc. For several reasons, mainly because of lack of workforce, OPATAM, OPAVAR and related developments were not included in the standard release of OPA. As a consequence, synchronisation of OPATAM with OPA’s releases could not be achieved on a regular basis, and all developments were on individual branches, without feedback to the OPATAM/OPAVAR system. The pool of potential users was reduced consequently. It is important not to repeat this error in the future, so as to ensure that NEMOTAM become a widely used community tool. A NEMOTAM working group was initiated in the framework of a CNRS/INSU/LEFE ASSIM2006 project, to investigate the feasibility of using TAPENADE (?), an AD tool, to speed up the writing of TAM for OPA9, the dynamical ocean component of NEMO. The goal of this working group was twofold. The first goal was to identify the strengths and weaknesses of TAPENADE by applying it directly to the NEMO source code, with as little human intervention as possible, in order to build a NEMOTAM prototype. The second goal was to define, based on the experience deriving the prototype, a strategy for developing a general-purpose NEMOTAM that both respects the NEMO code style/structure and gives acceptable computer performance (in terms of both CPU and memory requirements) for realistic ocean configurations. Providing feedback to the TAPENADE developers was an important aspect of this work to help improve future versions of the TAPENADE software. The results from this feasibility study demonstrated that TAPENADE was able to produce tangent-linear and adjoint models for NEMO, albeit for a fixed and somewhat simplified configuration (ORCA 2◦ with all non-differentiable options switched off (?)). Even for this simplified configuration, however, substantial human intervention and additional work was required to obtain a useable product from the raw TAPENADE-generated code. Three main drawbacks with TAPENADE were identified for this application. First, the memory management and CPU performance of the raw code were rather poor. Second, the current version of TAPENADE generates single-processor code only and cannot handle directives from the C-PreProcessor (CPP keys), which are widespread in NEMO. Third, the technique of binomial checkpointing that is used in TAPENADE to handle nonlinearities (see (?)) is not compatible, at least in its present implementation, with the incremental algorithm of NEMOVAR, which employs separate executables and (possibly) different resolutions for the outer and inner loops. Improved memory management and extensions to support MPP and CPP keys are planned in future versions of TAPENADE so the first two deficiencies are not fundamental. The third deficiency, however, is more problematic and it is likely that the trajectory management for nonlinearities in NEMOTAM will be done differently from TAPENADE, possibly along the lines of the simpler strategy implemented in OPAVAR. The modifications required to make TAPENADE or whatever other AD tool, compatible with the multi-incremental approach are really substantial and cannot be done in a short or medium term. Moreover the numerical performances of the TAPENADE generated TAM do not allow yet their use for ’big’ configurations and for operational applications. From that experience it has been decided to go toward the hand-coding approach, the use of AD tool being left aside for the time being. We may reconsider this in a medium or long term though. 6 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide VODA: ANR-08-COSI-016 1.2 1.2.1 Tangent and adjoint models (TAM) basics TAM, but what for ? The development of tangent and adjoint models is an important step in addressing sensitivity analysis and variational data assimilation problems in Oceanography. Sensitivity analysis is the study of how model output varies with changes in model inputs. The sensitivity information given by the adjoint model is used directly to gain an understanding of the physical processes. In data assimilation, one considers a cost function which is a measure of the model-data misfit. The adjoint sensitivities are used to build the gradient for descent algorithms. Similarly the tangent model is used in the context of the incremental algorithms to linearize the cost function around a background control. 1.2.2 Tangent and Adjoint coding principle The original program P , whatever its size and run time, computes a function F : X ∈ Rm → Y ∈ Rn which is the composition of the elementary functions computed by each run-time instruction. In other words if P executes a sequence of elementary statements Ik , k ∈ [1, .., p], then P actually evaluates F = fp ◦ fp−1 ◦ ... ◦ f1 where each fk is the function implemented bu Ik . Therefore one can apply the chain rule of derivative calculus to get the Jacobian matrix F 0 , i.e. the partial derivatives of each component of Y with respect to each component of X. Calling X0 = X and Xk = fk (Xk−1 ) the successive values of all intermediate variables, i.e. the successive states of the memory throughout execution of P , we get 0 F 0 (X) = fp0 (Xp−1 ) × fp−1 (Xp−2 ) × ... × f10 (X0 ) (1.1) The derivatives fk0 of each elementary instruction are easily built, and must be inserted in the differentiated program so that each of them has the values Xk−1 directly available for use. This process yields analytic derivatives, that are exact up to numerical accuracy. In practice, two sorts of derivatives are of particular importance in scientific computing: the tangent (or directional) derivatives, and the adjoint (or reverse) derivatives. The tangent derivative is the product dY = F 0 (X) × dX of the full Jacobian times a direction dX in the input space. From equation (1.1), we find 0 dY = F 0 (X) × dX = fp0 (Xp−1 ) × fp−1 (Xp−2 ) × ... × f10 (X0 ) × dX (1.2) which is most cheaply executed from right to left because matrix × vector products are much cheaper than matrix × matrix products. This is also the most convenient execution order because it uses the intermediate values Xk in the same order as the program P builds them. On the other Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide 7 VODA: ANR-08-COSI-016 hand the adjoint derivative is the product Xad = F 0? (X) × Yad of the transposed Jacobian times a weight vector Yad in the output space. The resulting Xad is the gradient of the dot product (Y.Yad ). From equation (1), we find 0? Xad = F 0? (X) × Yad = f10? (X0 ) × ... × fp−1 (Xp−2 ) × fp0? (Xp−1 ) × Yad (1.3) which is also most cheaply executed from right to left. However, this uses the intermediate values Xk in the inverse of their building order in P . For specific details about practical adjoint coding refer to (?). 1.2.3 Potential issues • the major approximations : non differentiability issues, simplification of the direct model before linearization, validity of the tangent linear hypothesis over some time window, etc. • the validation interface : classical tests for checking the correctness of tangent and adjoint models • numerical problems • numerical cost issues 1.3 Direct model changes Some modifications and additions are compulsory to integrate the Tangent and Adjoint Module to NEMO. They mainly consist in: • changing subroutines or parameters declaration from private to public for: – initialization – re-use • adding a subroutine to save the non-linear trajectory • adding parameters • updating namelist 1.3.1 Storing the non-linear trajectory To evaluate the tangent and/or the adjoint at a time t, some actual parameters of the direct model are needed at the corresponding time t. To fulfill this purpose a subroutine, tamtrj wri is added to write the trajectory. It is controlled with one logical parameter: ln trjwri. The trajectory is saved upon a given frequency ”nittrjfrq” only if the logical parameter is set to TRUE. New subroutine tam trj wri A new subdirectory TAM is created into OPA SRC named TAM. The file is tamtrj.F90. The purpose of this routine is to write on disk the model state trajectory for use with 4DVar. It is called in the opa model routine for the initializiation, and in the stp routine in order to perform the actual saving of the trajectory. The currently stored fields are listed in Table 1.1. 8 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide VODA: ANR-08-COSI-016 Parameters emp emps un vn tn sn ta sa tb sb avmu avmv aeiu aeiv aeiw uslp vslp wslpi wslpj avs strdmp hmlp Associated ccp keys lk traldf eiv lk traldf eiv lk traldf eiv key ldfslp key ldfslp key ldfslp key ldfslp key zdfddm key tradmp key tradmp Table 1.1: Fields stored in the reference trajectory Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide 9 VODA: ANR-08-COSI-016 Specific management of the initial state For the initial state of the model, we call the tam trj wri in the ’step’ routine just before the tracer block. We, then, have access to some parameters (tke and sbc for instance) as they are not yet computed in opa (outside ’step’). Linear interpolation For efficiency matters, the whole trajectory is of course not saved. Only a predefined subset of the time steps is saved. The frequency on which the trajectory is saved is controlled by namelist parameter nittrjfrq. The intermediate steps will be estimated in TAM by the mean of a linear interpolation. Namelist Parameters related to the model state saving is controlled by the namelist namtam with: • ln trjwri: Logical switch for writing out state trajectory • nittrjfrq: saving frequency of the trajectory 1.3.2 Modified routines in the direct code • opa.F90: – include initilialization of the module that handles the trajectory saving (USE tamtrj) • step.F90: – include calls for saving the trajectory (CALL tam trj wri) – add temporary variables (zta, zsa) to save ta/sa (this is a temporary fix because these variables are currently used as workspace in dynamic block) 1.3.3 Modified declarations in the direct code The following table describes the change from PRIVATE to PUBLIC of some routines and parameters. 1.4 1.4.1 Tangent linear model approximations Active variables degraded to passive Some active variables are treated as passive (mainly due to non-differentiability characteristics). • avmu, avmv: the vertical eddy viscosity. The turbulent closure scheme zdftke induces some non-differentiabilities for there variables. They are used in routine dynzdf_imp_tam. F90. 10 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide VODA: ANR-08-COSI-016 Modules dynadv daymod lib mpp par oce zdftke eosbn2 tradmp Routines dyn adv ctl day mth Parameters mpi comm opa jpni, jpnj, jpnij tke rst neos init cofdis, dtacof, dtacof zoom traqsr strdmp, ttrdmp, resto nksr, gdsr Table 1.2: List of modified declarations in the direct code NEMO • uslp, vslp: the slope of neutral surface (slope of isopycnal surfaces referenced locally) are treated as passive in order to not recomputed them at each step. They are used in routine traldf_iso_tam.F90. • wslpi/j: vertical mixing coefficient due to lateral mixing. They are used in routine trazdf_ imp_tam.F90. 1.4.2 Simplifications • removal of the upstream part of the second order centered scheme for traadv_cen2_tam. F90. It introduces some non-differentiabilities. Some complementary test must be held to fully implement the second order scheme • removal of zonal mean lateral diffusive heat and salt transport (pht ldf and pst ldf ) 1.5 Hand coding status The current status for the hand coding part of NEMOTAM / NEMOVAR is gathered in Tables 1.3 and 1.4. 1.6 Available options in NEMOTAM This sections contains information on current available configuration for NEMOTAM. To have an overview on available routines, please see the TAM development progress table. We use the following method: The physical choices are set through cpp keys and namelist parameters. We use below a similar table used for NEMO. See tables 1.5 and 1.6. Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide 11 VODA: ANR-08-COSI-016 Modules called by step tam Module(s) called by previous one Options Remarks Ocean dynamics (DYN) cla div tam.F90 cla tam.F90 divcur tam.F90 dynadv tam.F90 dynkeg dynzad dynadv dynadv tam.F90 tam.F90 cen2 tam.F90 ubs tam.F90 not in reference not in reference dynhpg tam.F90 hpg zco hpg zps hpg sco hpg hel hpg wdj, hpg djc, hpg rot dynldf tam.F90 dynldf dynldf dynldf dynldf lap tam.F90 bilap tam.F90 iso tam.F90 bilapg tam.F90 not in reference not in reference dynnxt tam.F90 lk vvl key obc, key bdy key agrif non-linear not in reference not in reference dynspg tam.F90 dynspg flt tam.F90 dynspg ts tam.F90 dynspg exp tam.F90 not in reference not in reference dynvor tam.F90 dyn dyn dyn dyn vor vor vor vor ens een ene mix not in reference not in reference dynzdf tam.F90 dynzdf exp tam.F90 dynzdf imp tam.F90 param. avmu and avmv as passive variables (if not: non-differentiability issue, see zdftke formulation) (should be treated as active var. when key zdf noncst is activated dynmem tam.F90 wzvmod tam.F90 Table 1.3: Hand coding status of NEMOTAM, Dynamics part 12 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide VODA: ANR-08-COSI-016 Modules called by step tam Module(s) called Options by previous one Ocean tracers (TRA) Remarks traadv tam.F90 no upstream scheme (equivalent to zcofi=0) not differenciable not differentiable not differentiable not differenciable not differentiable not differentiable traadv cen2 tam.F90 traadv traadv traadv traadv traadv traadv tvd tam.F90 muscl tam.F90 muscl2 tam.F90 ubs tam.F90 qck tam.F90 eiv tam.F90 trabbc tam.F90 hlmp is saved in direct model to handle uslp, vslp tradmp tam.F90 traldf tam.F90 key ldf ano uslp, vslp treated as passive not in reference not in reference pht ldf, pst ldf not include key obc, key bdy key agrif not in reference not in reference traldf bilap tam.F90 traldf bilapg tam.F90 traldf lap tam.F90 tranxt tam.F90 traqsr tam.F90 trasbc tam.F90 trazdf tam.F90 lk vvl non-linear wslpi, wslpj treated as passive trazdf imp tam.F90 lk vvl trazdf exp tam.F90 lk vvl Surface boundary conditions (SBC) sbcmod tam.F90 sbc sbc sbc sbc sbc sbc gyre tam.F90 flx tam.F90 ssr tam.F90 fwb tam.F90 clo tam.F90 ana tam.F90 Table 1.4: Hand coding status of NEMOTAM, Tracers and Surface Boundary Condition part Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide 13 VODA: ANR-08-COSI-016 Used for Horizontal Diffusion/Viscosity Functionalities CPP keys Lateral diffusion along isopycinal slopes Associated namelist logical key ldfslp Horizontal Tracer diffusion Tracer lateral diffusion eddy induced velocity parametrization Tracer lateral laplacian Iso-neutral application direction for operator Tracer lateral diffusion 1d=f(depth), 2d=f(lat,ion) or 3d=f(lat,ion, depth) Horizontal Dynamics Viscosity key traldf eiv ln traldf lap ln traldf iso key traldf c2d nam dynldf Dynamic Laplacian operator lateral viscosity Dynamic biLaplacian operator lateral viscosity Geopotential application direction viscosity Dynamic lateral viscosity 1d, 2d or 3d ln dynldf lap ln dynldf bilap ln dynldf lap key dynldf 3cd Vertical viscosity Diffusivity Vertical diffusion based on TKE closure scheme Vertical double diffusion mixing for salinity key zdftke namtke key zdfddm namddm Vertical convective processes enhanced vertical diffusion parametrization ln zdfevd namsbc, key forced daily, key tau daily Surface forcing fields Initial state and damping Internal Newtonian tracers (T,S) damping Use a 3D monthly salinity data (Levitus) Use a monthly temperature data (Levitus) Use a monthly sst data (Levitus) key tradmp key dtasal key dtatem key dtasst Sea surface temperature damping 14 Sea surface salinity damping Deliverable namdtp D1.3: NEMOTAM Reference namsbc, namsbc ssr, ln ssr namsbc, namsbc ssr, Manual ln ssr & User’s Guide Table 1.5: Available option in NEMOTAM, part1 VODA: ANR-08-COSI-016 Used for On-line diagnostics Functionalities Eddy induced velocity (Gent-Williams parametrization) Thermocline, 20◦ C-28◦ C isotherm depths Enable observation operator CPP keys Associated namelist logical key diaeiv key diahth key diaobs High Performances computing Enable MPP computing with MPI reproductibility key mpp mpi key mpi send key mpi send Vertical coordinates nam zgr ln zco ln zps Full steps - Z Partial steps - PS Pressure gradient treatment Filtered free surface algorithm key dynspg flt Vorticity scheme nam dynvor ln dynvor ens ln dynvor een nam traadv ln traadv cen2 enstrophy conserving energy and enstrophy conserving Advection schemes 2nd order centered Configuration Idealized Double Gyre configuration with a flat bottom Global ORCA configuration at 2deg resolution Open boundaries POMME configuration at 0.25◦ resolution key gyre key orca r2 key pomme r025 Other CPP keys Geothermal boundary condition flux Diffusive Bottom Boundary Layer key trabbl dif nambbc nambbl Table 1.6: Available option in NEMOTAM, part2 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide 15 VODA: ANR-08-COSI-016 1.7 1.7.1 The generic validation interface Introduction The validation interface aims at showing the consistency of the TAM with respect to the direct code. To fulfill this goal we demonstrate that: 1. the adjoint code is the adjoint of its tangent code 2. the tangent linear code is the tangent linear of the direct code with defined criteria. In general, TAM modules are subdivided into four parts: 1. 2. 3. 4. a tangent-linear routine an adjoint routine an adjoint test routine a tangent-linear routine Item 3 and 4 are optional. The strategy is to perform adjoint and tangent-linear tests for all critical sub-modules and the main module step tam for time-step integration. 1.7.2 Adjoint test By definition, A∗ (or tA in a finite-dimension case) is the adjoint operator of a linear operator A defined on a Hilbert space if it has the following property: (Ax, y) = (x, tAy) (1.4) In practice, we apply the comparison to random perturbations δx in the model-space and δy ∗ in the adjoint model-space. We have then: (Aδx, δy ∗ ) = (δx, tAδy ∗ ) (1.5) Adjoint test requirements • A saved trajectory of the direct model • An additionnal namelist (namtst for adjoint test, namtst tlm for tangent test Unitary test versus Global test Unitary test involves one or a limited set of routines. The pertubation vectors δX and δY are the input/output variables identified as actives variables. There is no time evolution. Global test involves time evolution (loop on step tam) and δX and δY are input/output variables identified as the control vector (or variables of main interest). 16 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide VODA: ANR-08-COSI-016 Test Pass Criteria We note (if exists) the relative error Err. The pass test criteria is based on the comparison of Err with the epsilon machine εmachine = eps. Err = (Aδx, δy ∗ ) − (δx, tAδy ∗ ) (Aδx, δy ∗ ) (1.6) The adjoint test results output is the file adjoint test.output xxx with the following label: Test status is OK test status is WARNING Test status is FAILED Err ≤ Eps Eps < Err ≤ 100 × Eps Err > 100 × Eps Table 1.7: Adjoint test labels definition Run the adjoint test • set the directory where the direct trajectory is stored • in the namelist namtst – – – – 1.7.3 set ln set ln set ln set ln tst tst tst tst nemotam to .true. (switch for M adjoint and tangent tests) cpd tam to .true. (switch for adjoint tests of the sub-components) stp tam to .true. (switch for adjoint tests of main components step tam) tan cpd and ln tst tan to .false. (switch for tangent tests) Tangent-Linear validation The tangent validation test checks that the Tangent Linear model L is a first-order approximation of the direct model M. Then, we have the following second-order Taylor expansion: M(X + pδX) = M(X) + pL(δX) + ε (1.7) When the factor p tends to zero: 1. Np = M(X + pδX) − M(X) tends to Lp = pL(δX) , first-order validation 2. ε behaves as O(p2 ) , second-order validation this means that for a given p0 , for p smaller than p0 the error of Tangent Linear with respect to the model is decreasing as p2 . Note that the second test (b) is not relevant if M is linear. Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide 17 VODA: ANR-08-COSI-016 Test a b Expression N orm(Np , .) N orm(Lp , .) N orm(Np − Lp , .) p2 behaviour 1 Cst Table 1.8: Tangent Linear tests. First-order check (a) and second-order check (b) Build the tangent test The tangent test is embedded into nemotam. To build it the user must add the compilation key key tst tlm in the CPP key set. Otherwise, a error message will raise during the run. Run the tangent test To run the tangent test the code must • set the directory where the direct trajectory is stored • in the namelist namtst – – – – set ln set ln set ln set ln tst to .true. tst nemotam to .true. (switch for M adjoint and tangent tests) tst tan cpd or ln tst tan to .true. (switch for tangent tests) tst cpd tam and ln tst stp tam to .false. (switch for adjoint tests) • in the namelist namtst tlm – set tlm bch to .true. (branching: ’0’ for M(X), ’1’ for M(X + pδX) and ’2’ for LpdX ) – set curr loop to .true. (current iteration) – set h ratio to .true. (value of the ratio factor p applied to δX) !----------------------------------------------------------------------! namtst_tlm tangent test parameters !----------------------------------------------------------------------! ! cur_loop = current loop iteration ! h_ratio = current h_ratio ! tlm_bch = flag for branching ! = 0 :M(X) , direct without perturbation ! = 1 :M(X+h_ratio.dX), direct with perturbation ! = 2 :L(h_ratio.dX) , linear &namtst_tlm tlm_bch = 1 cur_loop = 0 h_ratio = 10 / 18 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide VODA: ANR-08-COSI-016 Tangent test ouputs The output is saved with the name tan diag.output 0000. The header give information on the computed parameters. For iteration n, the output gives the follwing data: routine name, Nn = M(X+pn δX)−M(X), En = Nn /L(pn δX), Ern = (Nn −L(pn δX))/L(pn δX), L(pn δX), Nn − L(pn δX), (En − 1)/pn , Ern /pn . To have an overview of the behaviour of the tangent iteration can be made by the user changing the factor ratio pn . The factor takes usually a geometric progression as: pn = p0 r−n , with initial value p0 and common ratio r. 1.7.4 Tangent-Linear Hypothesis (HLT) validity The innovation vector validity relies on the validity of the Tangent Linear. Before any 4DVar assimilation we must insure that the tangent model is valid within our assimilation time-window. Otherwise the optimisation may fail as the cost function gradient will be degraded or even wrong. We define a maximum assimilation time-window Tamax . This time-window will be different for each configuration. According to each application the user its own criteria to define to define Tamax to fulfill its needs. The size of Tamax may vary from 1 to 3 months (???). To estimate Tamax we study the behaviour of y1 = Mt0 →t (x0 +δx) with respect to y2 = Mt0 →t (x0 )+ Lt0 →t (δx). The Tangent Linear will be valid as long as the difference y1 − y2 is ”small”. As discussed in (?) definition criteria of Tamax will rely on threshold on the RMS of y1 − y2 and the coefficient correlation ρy1 y2 . A correlation coefficient ρy1 y2 above 90% is considered as very good. RM S(y1 − y2 ) = v n uX u u (y1i − y2i )2 t i n , n: number of ocean points (1.8) n X (y1i − y¯1 ) · (y2i − y¯1 ) s n ρ y1 y2 = s n i X X (y1i − y¯1 )2 · (y2i − y¯2 )2 i , n: number of ocean points (1.9) i The initial condition as well as the increment δx are critical to define Tamax . The user should work with a typical increment δx of its configuration. HLT test tool NEMOTAM offers the possibility to analize the Tangent-Linear Hypothesis through a dedicate executable model hlt.exe. The increments δx is either given by the user either computed from two restarts file x0 and x1 (then δx = x0 − x1 ). To compute y1 and y2 , model hlt.exe should be run three times: two for the direct model trajectories (Mt0 →t (x0 + δx) and Mt0 →t (x0 )) and one for Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide 19 VODA: ANR-08-COSI-016 the tangent trajectory (Lt0 →t (δx)). It is driven through the namelists namhlt and tl tamtrj. !----------------------------------------------------------------------! namhlt tangent-linear hypothesis test parameters ! nstg flag for to compute model with or without perturbation ! 0: M(X) ! 1: M(X+dX) ! 2: L(dX) ! ln_hltt Logical switch for applying temperature increments ! ln_hlts Logical switch for applying salinity increments ! ln_hltuv Logical switch for applying horizontal velocity increments ! ln_hltssh Logical switch for applying sea surface height increments ! ln_incdx Logical swith to compute dx (F) or reading from file (T) ! ln_hnorm Logical swith to normalization (T) or not (F) using rhstdX ! rhstdt Upper bound of normalization factor for temperature ! rhstds Upper bound of normalization factor for salinity ! rhstduv Upper bound of normalization factor for velocity ! rhstdssh Upper bound of normalization factor for sea surface height !----------------------------------------------------------------------&namhlt nstg = 2 ln_hltt = .TRUE. ln_hlts = .TRUE. ln_hltuv = .TRUE. ln_hltssh = .FALSE. c_hltinc = "" ln_incdx = .FALSE. ln_hnorm = .TRUE. rhstdt = 1. rhstds = 0.1 rhstduv = 0.01 rhstdssh = 0.01 / !----------------------------------------------------------------------! namtl_trj tangent-linear trajectory frequency output ! ln_trjwri_tan Logical switch rajectory output ! nittrjfrq_tan frequency output for linear tangent !----------------------------------------------------------------------&namtl_trj ln_trjwri_tan = .TRUE. nittrjfrq_tan = 96 / 1.8 1.8.1 MPI parallelization General features We remind some basics for coding the corresponding adjoint of the tangent linear part: 20 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide VODA: ANR-08-COSI-016 Operation communication communication communication arithmetic inpout/Output inpout/Output Tangent-Linear send receive gather scatter write read Adjoint receive send scatter gather read write Table 1.9: ORCA2 Results of Adjoint test fo sub-routines There is no tangent of lbc lnk. The direct routine is used. For the adjoint part we use the coding rule described in table 1.9 1.8.2 Adjoint of MPI ALLGATHER From revision 3.2 of NEMO a new treatment of north fold horizontal bondary condition is performed using lbcnfd.F90 routine. This induces a MPI ALLGATHER usage in libmpp.F90 module (mpp lbc north 3d and mpp lbc north 2d). MPI ALLGATHER gathers data from all tasks and distribute it to all. Figure 1.1: MPI ALLGATHER task description (ref: http://rc.usf.edu/tutorials/classes /tutorial/mpi/chapter8.html If we call zloc Pi the data from task (i) and the zglo the result of MPI ALLGATHER and zloc adj Pi , zglo adj their adjoint equivalent. The adjoint counter-part of MPI_ALLGATHER(zglo, zloc) is Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide 21 VODA: ANR-08-COSI-016 MPI_SUM(zglo_adj) zloc__Pi = zglo_adj(:,i) As short demonstration, we consider a two-task case P1 and P2 . MPI ALLGATHER(zglo, zloc) is equivalent to: for P1 : (1) send and receive local data: send(zloc1_P1, P1, P2) ---> recv(zloc2_P1, P2, P1) ---> zloc1_P2 = zloc1_P1 zloc2_P1 = zloc2_P2 ( zlocK_Pj : local data K of j task) B- zglo operation zglo_P1(:,1) = zloc1_P1 zglo_P1(:,2) = zloc2_P1 we then write the adjoint as follows: zloc2_P1* zglo_P1*(:,2) zloc1_P1* zglo_P1*(:,1) = = = = zloc2_P1* + zglo_P1*(:,2) 0 zloc1_P1* + zglo_P1*(:,1) 0 zglo_tot* = zglo_P1* + zglo_P2* (i.e mpp_sum_nfd) (! because zloc_P1* is not null here, thus: zloc1_P1* = zloc_P1* + zloc1_P2*) zloc1_P1* zloc2_P1* 22 = zglo_tot*(:,1) = zglo_tot*(:,2) Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide Chapter 2 N EMOTAM U SER ’ S G UIDE 2.1 Source code structuration The project will work with three repositories (see Fig. 2.1b,c,d): • a NEMOBASE repository, with an official NEMO released version currently the tag nemo v3, available at : https://forge.ipsl.jussieu.fr/nemo/browser/tags/nemo_v3 • a NEMOVAR repository, with all developments related to the nemovar project, including new and modified files for NEMO, this is the standard version of NEMOVAR (currently tag N3.0-A2.0) • a VODA repository including new development for NEMOTAM and NEMOVAR within the VODA project (a) GIT tree (b) NEMOBASE (c) NEMOVAR Figure 2.1: Code organization 23 (d) VODA VODA: ANR-08-COSI-016 The building process will look for both repositories, if any redundancy is found, a VODA files will overwrite the corresponding NEMOVAR file, and a NEMOVAR file will overwrite the corresponding NEMOBASE file in the building folder. The NEMOBASE repository (see Fig. 2.1b) is the standard NEMO folder. The NEMOVAR repository (see Fig. 2.1c) is subdivided into 6 components: NEMO, NEMOQC, NEMOTAM, NEMOASSIM, UTIL and TOOL. The NEMOTAM folder is a mimic of the NEMO folder (meaning we have an OPATAM SRC folder subdivided into components as ASM, DOM, DYN, LDF, TRA, ...). The VODA folder (see Fig. 2.1d) mimic the NEMOVAR folder, with additional contributions. This organization gives a clear snapshot of what is being developped and lets some room for further development. 2.2 2.2.1 Getting started tutorial Installing the code using the NEMOVAR GIT repository Checking out the code from the GIT repository You need to ask the VODA project team to have developer access to the VODA project on the gforge.inria.fr server and access to the NEMO - IPSL. You can then follow the steps described below. Create first a new directory, where the source code will be installed. For instance, cd $VODA_GIT (=$HOME/VODA-PROJECT/VODA_GIT e.g.) ---> Install NEMOBASE go to https://forge.ipsl.jussieu.fr/nemo/browser/tags/nemo\_v3 (authentication required) and click on ’zip archive’ at the bottom of the page. put this archive in $VODA_GIT, unzip it and rename the obtained directory NEMOBASE (this name is mandatory) ---> Install NEMOVAR Download the following bundles, located on the gforge server under the "files" tab: NEMOVAR: N3.0-A2.0 => nemovar_N3.0-A2.0.bundle and put them in the $VODA_GIT/BUNDLES directory. In $VODA_GIT, you now have: tree BUNDLES/ BUNDLES/ |-- NEMOBASE.bundle.20090902 ‘-- nemovar_N3.0-A2.0.bundle Clone these bundles : git clone BUNDLES/NEMOBASE.bundle.20090902 NEMOBASE/ git clone BUNDLES/nemovar_N3.0-A2.0.bundle NEMOVAR/ 24 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide VODA: ANR-08-COSI-016 ---> Install VODA git clone git+ssh://[email protected]//gitroot/voda/voda.git VODA/ --> Final structure of the $VODA_GIT directory tree -d -L 2 $VODA_GIT VODA_GIT |-- BUNDLES |-- NEMOBASE | |-- AGRIF | |-- CONFIG | |-- CVSROOT | |-- DOC | |-- IOIPSL | |-- NEMO | |-- NVTK | ‘-- UTIL |-- NEMOVAR | |-- NEMO | |-- NEMOASSIM | |-- NEMOQC | |-- NEMOTAM | |-- TOOLS | ‘-- UTIL ‘-- VODA |-- NEMO |-- NEMOASSIM |-- NEMOTAM ‘-- UTIL Building the code (in default mode : ORCA2 validation interface) NEMOTAM makes use of the fcm based NEMOVAR compiling environment, a simple call to VODA/UTIL/build/fcmvmake.ksh with the usual arguments and the nemoall keyword should do the trick. > fcmvmake.ksh -c $COMPILE -t $WORKDIR -B nemoall the above requires that you create a VODA/UTIL/build/fcmconfig/bld/$COMPILE/nemo.cfg describing your compilation environment (examples are given in the same directory). How to run the code is described in section 2.3. 2.2.2 Installing the code using the NEMOTAM SVN repository Checking out the code from the SVN repository NEMOTAM can be extracted using svn like NEMO. People should first to register on NEMO website www.nemo-ocean.eu. Then NEMOTAM is available through modipsl framework, so modipsl has to be extracted first. Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide 25 VODA: ANR-08-COSI-016 • Extract modipsl: svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl • Extract NEMOTAM: cd modipsl/util ./model NEMOTAM Building the code (in default mode : ORCA2 validation interface) Choose the configuration to build. You can edit the script fait config tam to add a new one. cd modipsl/modeles/UTIL ./fait_config_tam ORCA2_TAM You can build the direct model in the same way: cd modipsl/modeles/UTIL ./fait_config ORCA2_LIM To create the Makefiles, edit first modipsl/util/AA make.gdef to find or add your compiler, then: cd modipsl/util ./ins_make To compile: cd modipsl/modeles/NEMOTAM/WORK gmake 2.2.3 Which executable for which test? To run a bunch of tests to verify your configuration you need to compile several executable according to your test target. The graph 2.2 give an overview of the relationship between the executable and the available tests. Running the code 2.3 2.3.1 Demonstrators The ORCA2 global configuration ORCA is the generic name given to global ocean configurations. Its specificity lies on the horizontal curvilinear mesh used to overcome the North Pole singularity found for geographical meshes. ORCA is based on a 2 degrees Mercator mesh, (i.e variation of meridian scale factor as cosinus of the latitude). In the northern hemisphere the mesh has two poles so that the ratio of anisotropy is nearly one everywhere. The vertical domain spreads from the surface to a depth of 5000m. There are 31 levels, with 10 levels in the to 100m. The time step depends on the resolution. It is 1h360 for ORCA2 so that there is 15 time steps in one day. 26 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide VODA: ANR-08-COSI-016 EXE file model tam.exe nemovar inner.exe Adjoint test OPATAM ln tst=T, ln tst nemotam=T ln tst cpd tam, ln tst stp tam Tangent test OPATAM ln tst=ln tst tan=T Adjoint test BKG, BAL, OBS ln tst=T, ln tst nemotam=T Gradient Test ln tst cpd tan ln tst obsadj, ln tst bkgadj ln tst=T, ln tst nemotam=T ln tst grd Single Obs. exp. model hlt.exe Linear Tangent Hypothesis Test ln sgl=T namhlt nstg=1,2,3 Figure 2.2: Releationships between executable files (first column) and available tests (second column). The thrid column focuses on the main namelist’s parameters (’namtst’ for adjoint, gradient, tangent tests, ’namobs’ for single obs.experiement and ’namhlt’ for linear tangent hypothesis) Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide 27 VODA: ANR-08-COSI-016 How to compile the ORCA2 configuration ? You must first follow the steps described in 2.2.1. The compilation process described below then relies upon the FCM (Flexible Configuration Manager) toolbox, which is included in VODA tree, see e.g. the following directory : ls $VODA_GIT/VODA/UTIL/fcm/bin Then, you may proceed in the following manner: • Get sample input data, needed for the model to run (http://ljk.imag.fr/membres/Arthur .Vidard/docs/nemov3_testdata.tar.gz) : • Customize the first configuration file nemo.cfg, which is located according to your platform architecture (mac intel here) : vi $VODA_GIT/VODA/UTIL/build/fcmconfig/bld/mac_intel/nemo.cfg • Customize the second configuration file, with CPP keys: cppkeys.ORCA2 Z31.cfg. A sample file is ready to be used, which you can modify: vi $VODA_GIT/VODA/UTIL/build/fcmconfig/nemo/cppkeys.ORCA2_Z31.cfg • You are now ready to compile the code. You need to create a ”scratch” directory (for example SCRATCH ORCA2 below), where compilation and execution will be performed: cd $VODA_GIT/VODA/UTIL/build ./fcmvmake.ksh -t $SCRATCH_ORCA2 -c mac_intel -B NEMO -G ORCA2_Z31 ./fcmvmake.ksh -t $SCRATCH_ORCA2 -c mac_intel -B NEMOTAM -G ORCA2_Z31 How to run the ORCA2 configuration ? A sample minimal script is given, just as an example : The direct model NEMO $VODA_GIT/VODA/UTIL/scripts/runnemo.ksh NEMO -g ORCA2_Z31 -t $SCRATCH_ORCA2 -A mac_intel -v The adjoint model NEMO tests $VODA_GIT/VODA/UTIL/scripts/runnemo.ksh TRAJ -v NEMOTAM -g ORCA2_Z31 -t $SCRATCH_ORCA2 -A mac_intel -j $ where $TRAJ is the reference trajectory directory. This script absolutely needs to be customized with respect to your working environment. Please have a look at this script before proceeding further. You will first need to create a reference trajectory by running only the direct model NEMO, and you then may want to use NEMOTAM to • launch the generic TAM test interface, • investigate the time span over which the tangent linear hypothethis remains valid, • etc. Testing the tangent model and the tangent linear hypothesis As discussed previously, the aims of the test is to illustrate the correctness of the tangent and how we have dealt with the non-linearity (if it exits) of the direct model. 28 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide VODA: ANR-08-COSI-016 Figure 2.3: Tangent test type: Four kinds of test results. Top left (a), linear routine with a residual following the error machine. Top right (b), non-linear routine with a residual decreasing as porder of 2. Bottom left (c), linear routine with a residual decreasing as p-order of 1. Bottom right (d), non-linear routine with a residual decreasing as p-order of 1. (Legend: Solid line is Np = M (x + pδx) − M (x), circle symbol Lp = L(pδx), triangle symbol Np − Lp ) Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide 29 VODA: ANR-08-COSI-016 Discussion We have divided the results into four categories (a), (b), (c) and (d) for analysis (see table 2.1 ). Category (a), illustrated by solsor tan in figure 2.3, gathers tangent-linear routines behaving as expected w.r.t to their direct routine (also linear) counter-part: the computed residual is close to the Lp error machine (the difference is around 10−15 . Category (b), illustrated by eos insitu tan Np − Lp in figure 2.3, gathers tangent-linear routines behaving as expected w.r.t to their direct routine (not linear) counter-part: the computed residual is equivalent to a O(p2 ). Categories (c) and (d) , illustrated by dynspg flt tan and tra adv cen2 tan in figure 2.3, gather suspicious cases we need to validate (or invalidate) through the choice made for the tangent-linear. Case (c) is for direct linear routine and (d) direct non-linear routine. Routine (L) solsor tan dynadv tan dynhpg tan traldf lap tan trazdf imp tan tra zpshde tan trasbc tan bn2 tan eos 1pt tan eos insitu2d tan eos insp tan eos insitu tan dynspg flt tan traadv cen2 flt tan Category class a b a b b or c ? b b b b b b b c d Table 2.1: ORCA2 Results of Tangent tests for sub-routines The case of traadv cen2 flt tan is fully understood as an approximation was done on the numerical scheme (the tangent linear does not include the upstream component of the numerical scheme). Below a certain threshold the upstream part (which is a O(p) in Np ) is no longer small compared to the second order scheme and thus, Lp . Testing the adjoint model The adjoint test for the ORCA2 configuration has proven satisfactory for the unitary routine tests (see table 2.2 ) and the global test (see tables 2.3 and 2.4. For the latter, the integration time tested varied from 1 to 15 days. 30 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide VODA: ANR-08-COSI-016 Routine (L) sol sor adj sbcfwb adj 1 1 sbcfwb adj 2 1 sbcfwb adj 1 2 sbcfwb adj 2 2 sbcfwb adj 1 3 sbcfwb adj 2 3 sbc ssr adj sbc gyre adj 1 sbc gyre adj 2 sbc ssm adj bn2 adj div cla adj div cur adj T1 div cur adj T2 dyn adv adj dynvor adj ens dynvor adj een dynldf adj lap dynldf adj blp dyn zdf adj dyn spg flt T1 dyn spg flt T2 dyn hpg adj dyn nxt adj wzv adj tra sbc adj tra qsr adj tra qsr adj tra dmp adj T1 tra dmp adj T2 tra dmp adj T3 tra adv cen2 tra cla adj Gi tra cla adj BM tra ldf iso ad trazdf exp adj trazdfimpadjT1 trazdfimpadjT2 trazdfimpadjT3 tra nxt adj eos adj insitu eos adj pot eos adj 2d eos adj 1pt zps hde adj istate tst (Lδx)| · W dy .525406844734452E+15 .711617836790924E+12 .711586863123878E+12 .721405227628168E+12 .721331456200554E+12 .711617836790924E+12 .711586863123878E+12 .289793337319649E+21 .575805928604001E+19 .575805928604001E+19 .585782123944269E+15 .763641537078732E+10 .134110702040363E+17 .307633707985862E+07 .321541201767843E+07 .252818496373574E+17 .252818508381207E+17 .252818506922870E+17 .311731217229964E+17 .252575821916828E+17 .252556994319522E+17 .790035995408866E+24 .315978990829171E+25 .253953510430215E+17 .758385105599429E+17 .258438017479678E+13 .517847966550809E+15 1 .134738569058199E+19 2 .134738569058199E+19 .152739703065678E+19 .152739703065678E+19 .152739703065678E+19 .152558770733990E+19 .487513953864278E+16 .158030397951644E+15 .152906929763072E+19 .235296147886893E+37 .208237542411001E+25 .208237542411001E+25 .208237542411001E+25 .405901725366058E+19 .405930021470975E+11 .241698506116296E+17 .249470161251995E+09 .108202320860649E+10 .417898655654515E+18 .140086193684597E+19 δdx| · L| W δy .525406844734452E+15 .711617836790924E+12 .711586863123878E+12 .721405227628168E+12 .721331456200554E+12 .711617836790924E+12 .711586863123878E+12 .289793337319649E+21 .575805928604001E+19 .575805928604001E+19 .585782123944269E+15 .763641537078733E+10 .134110702040363E+17 .307633707985862E+07 .321541201767843E+07 .252818496373574E+17 .252818508381207E+17 .252818506922870E+17 .311731217229965E+17 .252575821916828E+17 .252556994319522E+17 .790035995408866E+24 .315978990829171E+25 .253953510430214E+17 .758385105599429E+17 .258438017479678E+13 .517847966550810E+15 .134738569058200E+19 .134738569058200E+19 .152739703065678E+19 .152739703065678E+19 .152739703065678E+19 .152558770733990E+19 .487513953864278E+16 .158030397951644E+15 .152906929763072E+19 .235296147886893E+37 .208237542411001E+25 .208237542411001E+25 .208237542411001E+25 .405901725366058E+19 .405930021470975E+11 .241698506116297E+17 .249470161251995E+09 .108202320860649E+10 .417898655654515E+18 .140086193684597E+19 Rel. Err. .0E+00 .7E-15 .0E+00 .0E+00 .0E+00 .7E-15 .0E+00 .2E-15 .0E+00 .0E+00 .0E+00 .5E-15 .1E-15 .0E+00 .6E-15 .2E-15 .0E+00 .0E+00 .3E-15 .2E-15 .0E+00 .0E+00 .3E-15 .3E-15 .0E+00 .0E+00 .5E-15 .1E-14 .1E-14 .0E+00 .0E+00 .0E+00 .1E-14 .2E-15 .6E-15 .0E+00 .6E-15 .5E-15 .5E-15 .5E-15 .9E-15 .0E+00 .5E-15 .4E-15 .2E-15 .2E-15 .7E-15 Mach. eps. .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 Status ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok ok Table 2.2: ORCA2 Results of the Adjoint tests for the main sub-routines Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide 31 VODA: ANR-08-COSI-016 Routine (L) step adj U step adj V step adj T step adj S step adj SSH step adj (Lδx)| · W dy .161275281471349E+18 .318648865958325E+18 .132120109179127E+19 .281362832449564E+18 .509250348807475E+10 .206971044341451E+19 δdx| · L| W δy .161275281471349E+18 .318648865958325E+18 .132120109179127E+19 .281362832449564E+18 .509250348807474E+10 .206971044341451E+19 Rel. Err. .6E-15 .4E-15 .0E+00 .0E+00 .2E-14 .1E-15 Mach. eps. .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 Status ok ok ok ok ok ok Table 2.3: ORCA2 Results of the Adjoint tests for the main routine step tam for a run-window of one day. From line one to four, each control parameter is perturbed individually. Column six, all control parameter are perturbed. Routine (L) step adj U step adj V step adj T step adj S step adj SSH step adj (Lδx)| · W dy .102643067009108E+18 .167827302823834E+18 .893406215846414E+18 .220114453479612E+18 .265003707314821E+10 .137645166426475E+19 δdx| · L| W δy .102643067009108E+18 .167827302823834E+18 .893406215846413E+18 .220114453479612E+18 .265003707314821E+10 .137645166426475E+19 Rel. Err. .2E-15 .4E-15 .7E-15 .4E-15 .4E-15 .6E-15 Mach. eps. .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 .2E-14 Status ok ok ok ok ok ok Table 2.4: ORCA2 Results of the Adjoint tests for the main routine step tam for a run-window of five days. From line one to four, each control parameter is perturbed individually. Column six, all control parameter are perturbed. 2.3.2 The GYRE regional configuration GYRE is an idealized configuration representing double gyres in the Northern hemisphere. It is β-plane with a regular grid spacing at 1 degree horizontal resolution. There is 31 verticles levels. The configuration is forced with analytical heat, freswater and wind-stress fields. How to compile the GYRE configuration ? Similar to ORCA2 configuration replacing ORCA2 Z31 by GYRE Z31. How to run the GYRE configuration ? Similar to ORCA2 configuration replacing ORCA2 Z31 by GYRE Z31. Tangent-Linear Hypothesis (HLT) validity To illustrate the tangent linear hypothesis validity test We use an initial state x0 ten years from spinup and an increment δx computed from an previous assimiilation with (|δxT | ≤ 1 , |δxS | ≤ 0.1, |δxU | ≤ 0.0022 and |δxV | ≤ 0.0019) . see figure 2.4 here: place more stats on δx (mean, min, max, std) ? here: add comments for the graph 32 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide VODA: ANR-08-COSI-016 Figure 2.4: Tangent linear hypothsesis validty for GYRE Z31: plot in function of time-window in days over 1 year. Top graphs are RMS of T, S, U and V variables in S.I units. Bottom graphs are related coefficient correlation. 2.3.3 The POMME configuration (toy model for handling open boundaries) Description of the configuration NEMO is distributed with some reference configurations allowing the users to set up a first application, and the developers to validate their developments. POMME is a small square box configuration (30 × 40 × 46 grid points), extracted from NEMO DRAKKAR / ORCA025. All boundaries are open ocean boundaries (including the corners). This configuration was set up to facilitate developments and tests for regional configurations (however, it is not yet an official NEMO reference configuration). It should be soon included in the NVTK validation toolkit. How to compile the POMME configuration ? You must first follow the steps described in 2.2.1. The compilation process described below then relies upon the FCM (Flexible Configuration Manager) toolbox, which is included in VODA tree, see e.g. the following directory : ls $VODA_GIT/VODA/UTIL/fcm/bin Then, you may proceed in the following manner: • Get sample input data, needed for the model to run: wget http://www.locean-ipsl.upmc.fr/˜cdlod/data/data_POMME_Z46.tar.gz Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide 33 VODA: ANR-08-COSI-016 • Customize the first configuration file nemo.cfg, which is located according to your platform architecture (mac intel here) : vi $VODA_GIT/VODA/UTIL/build/fcmconfig/bld/mac_intel/nemo.cfg • Customize the second configuration file, with CPP keys: cppkeys.POMME Z46.cfg. A sample file is ready to be used, which you can modify: vi $VODA_GIT/VODA/UTIL/build/fcmconfig/nemo/cppkeys.POMME_Z46.cfg • You are now ready to compile the code. You need to create a ”scratch” directory (for example SCRATCH POMME below), where compilation and execution will be performed: cd $VODA_GIT/VODA/UTIL/build ./fcmvmake.ksh -t $SCRATCH_POMME -c mac_intel -B NEMO -G POMME_Z46 ./fcmvmake.ksh -t $SCRATCH_POMME -c mac_intel -B NEMOTAM -G POMME_Z46 How to run the POMME configuration ? A sample minimal script is given, just as an example : vi $VODA_GIT/VODA/UTIL/scripts/run_pomme.ksh This script absolutely needs to be customized with respect to your working environment. Please have a look at this script before proceeding further. You will first need to create a reference trajectory by running only the direct model NEMO, and you then may want to use NEMOTAM to • launch the generic TAM test interface, • investigate the time span over which the tangent linear hypothethis remains valid, • etc. Testing the tangent model and the tangent linear hypothesis coming soon Testing the adjoint model coming soon 2.4 2.4.1 How-to update NEMOTAM with your contribution? Coding rules for TAM NEMOTAM modules should follow the NEMO convention. However additional rules has to be followed: • For a given direct module mod.F90 (that contains the subroutine sub), a corresponding module mod tam.F90 has to be created and it should contain the tangent linear subroutine called sub tan, the adjoint subroutine sub adj and the corresponding testing routine sub adj tst. 34 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide VODA: ANR-08-COSI-016 • For a given active direct variable var, the corresponding tangent linear and adjoint variable should be named var tl and var ad respectively. The underscore is omitted for local active variable. To summarized, active variables un, pun, zun will give un tl, pun tl and zun tl in the tangent module • For historical reasons, some modules do not follow these rules, feel free to update them. If the TAM module is a direct translation of the corresponding OPATAM routine, a good practice is to account for it in the first line of the history (i.i stating that the 8.2 version was done by e.g Huey, Dewey, and Louie) 2.4.2 Workflow for introducing additional developments coming soon Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide 35 VODA: ANR-08-COSI-016 2.5 Frequently asked questions 2.5.1 I can’t use GDB after compiling with FCM When I run the code with GDB for debugging purposes, the library lib fcm nemotam.a is not found, and thus gdb does not have access to symbolic informations associated with the modules of the code : we can’t set breakpoints, we can’t look for the content of variables, etc. This library is a temporary one generated by FCM during the build process, but it is removed once the compilation is finished. This problem only appears on Mac OSX’s computers. For unknown reasons, the linker looses track when going from fcm’s temporary libraries to the final libraries. The temporary workaround is to recreate these temporary libraries from the object files (but you will have to redo it each time you compile): cd $SCRATCH_POMME/fcmbuild/POMME_Z46_mac_intel.1.1/build/lib/ ar -q lib__fcm__nemotam.a ../obj/*.o 2.5.2 I have many commas in the namelist generated by piano This happens if your config namelist have lines ending with a comma. This feature is accepted by FORTRAN, but the PIANO python script parser do not handle this. Removing the commas at the end of each line will solve the problem. 2.5.3 I have negative values in my cost function terms See congrad.F90 : Compute cost for diagnostic purposes only pf=zf0+0.5*DOT_PRODUCT(...) It is this term in congrad which appears to be < 0 and is written to the cost file, whereas we would expect to write the pcost from simvar.F90. Under investigation 2.5.4 My run stops with ’PROBLEM WITH THE MATRIX OR WITH THE PRECONDITIONER’ on stderr This problem was reported in the POMME and GYRE configurations. First, it is worth noting that this message disappears if you set ln_tst_grad = .TRUE. which is absolutely not a normal behaviour. Needs further investigation. The problem is solved by increasing the namelist nmin parameter. For example, change nmin = 300 ! minimum of iterations for the SOR solver = 500 ! minimum of iterations for the SOR solver into : nmin 36 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide VODA: ANR-08-COSI-016 in namsol. This behaviour is not normal, it suggests a convergence problem in the tangent barotropic solver. Needs further investigation. Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide 37 VODA: ANR-08-COSI-016 38 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide List of Figures 1.1 MPI ALLGATHER task description (ref: http://rc.usf.edu/tutorials/classes /tutorial/mpi/chapter8.html . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.1 2.2 Code organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Releationships between executable files (first column) and available tests (second column). The thrid column focuses on the main namelist’s parameters (’namtst’ for adjoint, gradient, tangent tests, ’namobs’ for single obs.experiement and ’namhlt’ for linear tangent hypothesis) . . . . . . . . . . . . . . . . . . . . . . . . . . . Tangent test type: Four kinds of test results. Top left (a), linear routine with a residual following the error machine. Top right (b), non-linear routine with a residual decreasing as p-order of 2. Bottom left (c), linear routine with a residual decreasing as p-order of 1. Bottom right (d), non-linear routine with a residual decreasing as p-order of 1. (Legend: Solid line is Np = M (x + pδx) − M (x), circle symbol Lp = L(pδx), triangle symbol Np − Lp ) . . . . . . . . . . . . . . . . . . . . . . Tangent linear hypothsesis validty for GYRE Z31: plot in function of timewindow in days over 1 year. Top graphs are RMS of T, S, U and V variables in S.I units. Bottom graphs are related coefficient correlation. . . . . . . . . . . 2.3 2.4 39 . 23 . 27 . 29 . 33 VODA: ANR-08-COSI-016 40 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide List of Tables 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 Fields stored in the reference trajectory . . . . . . . . . . . . . . . . . . . . . . . List of modified declarations in the direct code NEMO . . . . . . . . . . . . . . Hand coding status of NEMOTAM, Dynamics part . . . . . . . . . . . . . . . . Hand coding status of NEMOTAM, Tracers and Surface Boundary Condition part Available option in NEMOTAM, part1 . . . . . . . . . . . . . . . . . . . . . . . Available option in NEMOTAM, part2 . . . . . . . . . . . . . . . . . . . . . . . Adjoint test labels definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tangent Linear tests. First-order check (a) and second-order check (b) . . . . . . ORCA2 Results of Adjoint test fo sub-routines . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 2.2 2.3 ORCA2 Results of Tangent tests for sub-routines . . . . . . . . . . . . . . . . . ORCA2 Results of the Adjoint tests for the main sub-routines . . . . . . . . . . . ORCA2 Results of the Adjoint tests for the main routine step tam for a run-window of one day. From line one to four, each control parameter is perturbed individually. Column six, all control parameter are perturbed. . . . . . . . . . . . . . . . . . ORCA2 Results of the Adjoint tests for the main routine step tam for a run-window of five days. From line one to four, each control parameter is perturbed individually. Column six, all control parameter are perturbed. . . . . . . . . . . . . . . . . 30 . 31 2.4 41 9 11 12 13 14 15 17 18 21 . 32 . 32 VODA: ANR-08-COSI-016 APPENDIX : Acronyms • AD : Automatic differentiation • AGRIF : Adaptive Grid Refinement In Fortran. Outil de raffinement de maillage et de nesting d´evelopp´e au sein de MOISE et inclus dans NEMO. • ESOPA : Equipe Syst`eme OPA, qui g´erait les d´eveloppements de OPA. A e´ t´e remplac´ee par la NEMO-Team apr`es la mise en place du consortium NEMO • MITgcm : Mod`ele de circulation g´en´eral pour l’oc´ean et l’atmosph`ere d´evelopp´e au MIT • NEMO : Nucleus for European Modelling of the Ocean. • NEMOTAM : TAM pour NEMO • NEMO-Team : Equipe g´erant les d´eveloppements de NEMO • NEMOVAR : syst`eme d’assimilation variationnelle de donn´ees pour NEMO • OPA : Ocean Parall´elis´e. Le mod`ele direct d’oc´ean, composant de NEMO. • OPATAM : TAM pour OPA 8.x • OPAVAR : syst`eme d’assimilation variationnelle de donn´es pour OPA 8.x • ORCA : grille globale pour OPA • PONGO : Pˆole d’Oc´ean Num´erique GrenOblois, regroupant les e´ quipes MOISE (LJK/INRIA) et MEOM (LEGI) • ROMS : Regional Ocean Modelling System • TAF : Transformation of Algorithms in Fortran. G´en´erateur automatique de code tangent et adjoint utilis´e pour MITgcm. • TAM : Mod`eles Tangent et Adjoint • TAPENADE : G´en´erateur automatique de code tangent et adjoint d´evelopp´e par le projet TROPICS de l’INRIA 42 Deliverable D1.3: NEMOTAM Reference Manual & User’s Guide

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Download PDF

advertisement