WorldWideScience

Sample records for nonlinear markov process

  1. Nonlinearly perturbed semi-Markov processes

    CERN Document Server

    Silvestrov, Dmitrii

    2017-01-01

    The book presents new methods of asymptotic analysis for nonlinearly perturbed semi-Markov processes with a finite phase space. These methods are based on special time-space screening procedures for sequential phase space reduction of semi-Markov processes combined with the systematical use of operational calculus for Laurent asymptotic expansions. Effective recurrent algorithms are composed for getting asymptotic expansions, without and with explicit upper bounds for remainders, for power moments of hitting times, stationary and conditional quasi-stationary distributions for nonlinearly perturbed semi-Markov processes. These results are illustrated by asymptotic expansions for birth-death-type semi-Markov processes, which play an important role in various applications. The book will be a useful contribution to the continuing intensive studies in the area. It is an essential reference for theoretical and applied researchers in the field of stochastic processes and their applications that will cont...

  2. Generalized Mass Action Law and Thermodynamics of Nonlinear Markov Processes

    CERN Document Server

    Gorban, A N

    2015-01-01

    The nonlinear Markov processes are the measure-valued dynamical systems which preserve positivity. They can be represented as the law of large numbers limits of general Markov models of interacting particles. In physics, the kinetic equations allow Lyapunov functionals (entropy, free energy, etc.). This may be considered as a sort of inheritance of the Lyapunov functionals from the microscopic master equations. We study nonlinear Markov processes that inherit thermodynamic properties from the microscopic linear Markov processes. We develop the thermodynamics of nonlinear Markov processes and analyze the asymptotic assumption, which are sufficient for this inheritance.

  3. Nonlinear Markov Control Processes and Games

    Science.gov (United States)

    2012-11-15

    further research we indicated possible extensions to state spaces with nontrivial geometry, to the controlled nonlinear quantum dynamic semigroups and...space nonlinear Markov semigroup is a one-parameter semigroup of (possibly nonlinear) transformations of the unit simplex in n-dimensional Euclidean...certain mixing property of nonlinear transition probabilities. In case of the semigroup parametrized by continuous time one defines its generator as the

  4. Recombination Processes and Nonlinear Markov Chains.

    Science.gov (United States)

    Pirogov, Sergey; Rybko, Alexander; Kalinina, Anastasia; Gelfand, Mikhail

    2016-09-01

    Bacteria are known to exchange genetic information by horizontal gene transfer. Since the frequency of homologous recombination depends on the similarity between the recombining segments, several studies examined whether this could lead to the emergence of subspecies. Most of them simulated fixed-size Wright-Fisher populations, in which the genetic drift should be taken into account. Here, we use nonlinear Markov processes to describe a bacterial population evolving under mutation and recombination. We consider a population structure as a probability measure on the space of genomes. This approach implies the infinite population size limit, and thus, the genetic drift is not assumed. We prove that under these conditions, the emergence of subspecies is impossible.

  5. System Identification and Filtering of Nonlinear Controlled Markov Processes by Canonical Variate Analysis

    Science.gov (United States)

    1989-10-30

    In this Phase I SBIR study, new methods are developed for the system identification and stochastic filtering of nonlinear controlled Markov processes...state space Markov process models and canonical variate analysis (CVA) for obtaining optimal nonlinear procedures for system identification and stochastic

  6. Markov Chains and Markov Processes

    OpenAIRE

    2016-01-01

    Markov chain, which was named after Andrew Markov is a mathematical system that transfers a state to another state. Many real world systems contain uncertainty. This study helps us to understand the basic idea of a Markov chain and how is been useful in our daily lives. For some times there had been suspense on distinct predictions and future existences. Also in different games there had been different expectations or results involved. That is the reason why we need Markov chains to predict o...

  7. Markov processes and controlled Markov chains

    CERN Document Server

    Filar, Jerzy; Chen, Anyue

    2002-01-01

    The general theory of stochastic processes and the more specialized theory of Markov processes evolved enormously in the second half of the last century. In parallel, the theory of controlled Markov chains (or Markov decision processes) was being pioneered by control engineers and operations researchers. Researchers in Markov processes and controlled Markov chains have been, for a long time, aware of the synergies between these two subject areas. However, this may be the first volume dedicated to highlighting these synergies and, almost certainly, it is the first volume that emphasizes the contributions of the vibrant and growing Chinese school of probability. The chapters that appear in this book reflect both the maturity and the vitality of modern day Markov processes and controlled Markov chains. They also will provide an opportunity to trace the connections that have emerged between the work done by members of the Chinese school of probability and the work done by the European, US, Central and South Ameri...

  8. Semi-Markov processes

    CERN Document Server

    Grabski

    2014-01-01

    Semi-Markov Processes: Applications in System Reliability and Maintenance is a modern view of discrete state space and continuous time semi-Markov processes and their applications in reliability and maintenance. The book explains how to construct semi-Markov models and discusses the different reliability parameters and characteristics that can be obtained from those models. The book is a useful resource for mathematicians, engineering practitioners, and PhD and MSc students who want to understand the basic concepts and results of semi-Markov process theory. Clearly defines the properties and

  9. NONLINEAR EXPECTATIONS AND NONLINEAR MARKOV CHAINS

    Institute of Scientific and Technical Information of China (English)

    PENG SHIGE

    2005-01-01

    This paper deals with nonlinear expectations. The author obtains a nonlinear generalization of the well-known Kolmogorov's consistent theorem and then use it to construct filtration-consistent nonlinear expectations via nonlinear Markov chains. Compared to the author's previous results, i.e., the theory of g-expectations introduced via BSDE on a probability space, the present framework is not based on a given probability measure. Many fully nonlinear and singular situations are covered. The induced topology is a natural generalization of Lp-norms and L∞-norm in linear situations.The author also obtains the existence and uniqueness result of BSDE under this new framework and develops a nonlinear type of von Neumann-Morgenstern representation theorem to utilities and present dynamic risk measures.

  10. Markov reward processes

    Science.gov (United States)

    Smith, R. M.

    1991-01-01

    Numerous applications in the area of computer system analysis can be effectively studied with Markov reward models. These models describe the behavior of the system with a continuous-time Markov chain, where a reward rate is associated with each state. In a reliability/availability model, upstates may have reward rate 1 and down states may have reward rate zero associated with them. In a queueing model, the number of jobs of certain type in a given state may be the reward rate attached to that state. In a combined model of performance and reliability, the reward rate of a state may be the computational capacity, or a related performance measure. Expected steady-state reward rate and expected instantaneous reward rate are clearly useful measures of the Markov reward model. More generally, the distribution of accumulated reward or time-averaged reward over a finite time interval may be determined from the solution of the Markov reward model. This information is of great practical significance in situations where the workload can be well characterized (deterministically, or by continuous functions e.g., distributions). The design process in the development of a computer system is an expensive and long term endeavor. For aerospace applications the reliability of the computer system is essential, as is the ability to complete critical workloads in a well defined real time interval. Consequently, effective modeling of such systems must take into account both performance and reliability. This fact motivates our use of Markov reward models to aid in the development and evaluation of fault tolerant computer systems.

  11. Process Algebra and Markov Chains

    NARCIS (Netherlands)

    Brinksma, Hendrik; Hermanns, H.; Brinksma, Hendrik; Hermanns, H.; Katoen, Joost P.

    This paper surveys and relates the basic concepts of process algebra and the modelling of continuous time Markov chains. It provides basic introductions to both fields, where we also study the Markov chains from an algebraic perspective, viz. that of Markov chain algebra. We then proceed to study

  12. On nonlinear Markov chain Monte Carlo

    CERN Document Server

    Andrieu, Christophe; Doucet, Arnaud; Del Moral, Pierre; 10.3150/10-BEJ307

    2011-01-01

    Let $\\mathscr{P}(E)$ be the space of probability measures on a measurable space $(E,\\mathcal{E})$. In this paper we introduce a class of nonlinear Markov chain Monte Carlo (MCMC) methods for simulating from a probability measure $\\pi\\in\\mathscr{P}(E)$. Nonlinear Markov kernels (see [Feynman--Kac Formulae: Genealogical and Interacting Particle Systems with Applications (2004) Springer]) $K:\\mathscr{P}(E)\\times E\\rightarrow\\mathscr{P}(E)$ can be constructed to, in some sense, improve over MCMC methods. However, such nonlinear kernels cannot be simulated exactly, so approximations of the nonlinear kernels are constructed using auxiliary or potentially self-interacting chains. Several nonlinear kernels are presented and it is demonstrated that, under some conditions, the associated approximations exhibit a strong law of large numbers; our proof technique is via the Poisson equation and Foster--Lyapunov conditions. We investigate the performance of our approximations with some simulations.

  13. Markov processes, semigroups and generators

    CERN Document Server

    Kolokoltsov, Vassili N

    2011-01-01

    This work offers a highly useful, well developed reference on Markov processes, the universal model for random processes and evolutions. The wide range of applications, in exact sciences as well as in other areas like social studies, require a volume that offers a refresher on fundamentals before conveying the Markov processes and examples for applications. This work does just that, and with the necessary mathematical rigor.

  14. Markov Chains and Chemical Processes

    Science.gov (United States)

    Miller, P. J.

    1972-01-01

    Views as important the relating of abstract ideas of modern mathematics now being taught in the schools to situations encountered in the sciences. Describes use of matrices and Markov chains to study first-order processes. (Author/DF)

  15. Markov processes characterization and convergence

    CERN Document Server

    Ethier, Stewart N

    2009-01-01

    The Wiley-Interscience Paperback Series consists of selected books that have been made more accessible to consumers in an effort to increase global appeal and general circulation. With these new unabridged softcover volumes, Wiley hopes to extend the lives of these works by making them available to future generations of statisticians, mathematicians, and scientists."[A]nyone who works with Markov processes whose state space is uncountably infinite will need this most impressive book as a guide and reference."-American Scientist"There is no question but that space should immediately be reserved for [this] book on the library shelf. Those who aspire to mastery of the contents should also reserve a large number of long winter evenings."-Zentralblatt f?r Mathematik und ihre Grenzgebiete/Mathematics Abstracts"Ethier and Kurtz have produced an excellent treatment of the modern theory of Markov processes that [is] useful both as a reference work and as a graduate textbook."-Journal of Statistical PhysicsMarkov Proce...

  16. Green functions and Langevin equations for nonlinear diffusion equations: A comment on ‘Markov processes, Hurst exponents, and nonlinear diffusion equations’ by Bassler et al.

    Science.gov (United States)

    Frank, T. D.

    2008-02-01

    We discuss two central claims made in the study by Bassler et al. [K.E. Bassler, G.H. Gunaratne, J.L. McCauley, Physica A 369 (2006) 343]. Bassler et al. claimed that Green functions and Langevin equations cannot be defined for nonlinear diffusion equations. In addition, they claimed that nonlinear diffusion equations are linear partial differential equations disguised as nonlinear ones. We review bottom-up and top-down approaches that have been used in the literature to derive Green functions for nonlinear diffusion equations and, in doing so, show that the first claim needs to be revised. We show that the second claim as well needs to be revised. To this end, we point out similarities and differences between non-autonomous linear Fokker-Planck equations and autonomous nonlinear Fokker-Planck equations. In this context, we raise the question whether Bassler et al.’s approach to financial markets is physically plausible because it necessitates the introduction of external traders and causes. Such external entities can easily be eliminated when taking self-organization principles and concepts of nonextensive thermostatistics into account and modeling financial processes by means of nonlinear Fokker-Planck equations.

  17. Markov process functionals in finance and insurance

    Institute of Scientific and Technical Information of China (English)

    GENG Xian-min; LI Liang

    2009-01-01

    The Maxkov property of Maxkov process functionals which axe frequently used in economy, finance, engineering and statistic analysis is studied. The conditions to judge Maxkov property of some important Markov process functionals axe presented, the following conclusions are obtained: the multidimensional process with independent increments is a multidimensional Markov process; the functional in the form of path integral of process with independent incre-ments is a Markov process; the surplus process with the doubly stochastic Poisson process is a vector Markov process. The conditions for linear transformation of vector Maxkov process being still a Maxkov process are given.

  18. Maximizing Entropy over Markov Processes

    DEFF Research Database (Denmark)

    Biondi, Fabrizio; Legay, Axel; Nielsen, Bo Friis

    2013-01-01

    computation reduces to finding a model of a specification with highest entropy. Entropy maximization for probabilistic process specifications has not been studied before, even though it is well known in Bayesian inference for discrete distributions. We give a characterization of global entropy of a process...... as a reward function, a polynomial algorithm to verify the existence of an system maximizing entropy among those respecting a specification, a procedure for the maximization of reward functions over Interval Markov Chains and its application to synthesize an implementation maximizing entropy. We show how...

  19. Maximizing entropy over Markov processes

    DEFF Research Database (Denmark)

    Biondi, Fabrizio; Legay, Axel; Nielsen, Bo Friis

    2014-01-01

    computation reduces to finding a model of a specification with highest entropy. Entropy maximization for probabilistic process specifications has not been studied before, even though it is well known in Bayesian inference for discrete distributions. We give a characterization of global entropy of a process...... as a reward function, a polynomial algorithm to verify the existence of a system maximizing entropy among those respecting a specification, a procedure for the maximization of reward functions over Interval Markov Chains and its application to synthesize an implementation maximizing entropy. We show how...

  20. A compositional framework for Markov processes

    Science.gov (United States)

    Baez, John C.; Fong, Brendan; Pollard, Blake S.

    2016-03-01

    We define the concept of an "open" Markov process, or more precisely, continuous-time Markov chain, which is one where probability can flow in or out of certain states called "inputs" and "outputs." One can build up a Markov process from smaller open pieces. This process is formalized by making open Markov processes into the morphisms of a dagger compact category. We show that the behavior of a detailed balanced open Markov process is determined by a principle of minimum dissipation, closely related to Prigogine's principle of minimum entropy production. Using this fact, we set up a functor mapping open detailed balanced Markov processes to open circuits made of linear resistors. We also describe how to "black box" an open Markov process, obtaining the linear relation between input and output data that holds in any steady state, including nonequilibrium steady states with a nonzero flow of probability through the system. We prove that black boxing gives a symmetric monoidal dagger functor sending open detailed balanced Markov processes to Lagrangian relations between symplectic vector spaces. This allows us to compute the steady state behavior of an open detailed balanced Markov process from the behaviors of smaller pieces from which it is built. We relate this black box functor to a previously constructed black box functor for circuits.

  1. Inhomogeneous Markov point processes by transformation

    DEFF Research Database (Denmark)

    Jensen, Eva B. Vedel; Nielsen, Linda Stougaard

    2000-01-01

    We construct parametrized models for point processes, allowing for both inhomogeneity and interaction. The inhomogeneity is obtained by applying parametrized transformations to homogeneous Markov point processes. An interesting model class, which can be constructed by this transformation approach......, is that of exponential inhomogeneous Markov point processes. Statistical inference For such processes is discussed in some detail....

  2. NonMarkov Ito Processes with 1- state memory

    Science.gov (United States)

    McCauley, Joseph L.

    2010-08-01

    A Markov process, by definition, cannot depend on any previous state other than the last observed state. An Ito process implies the Fokker-Planck and Kolmogorov backward time partial differential eqns. for transition densities, which in turn imply the Chapman-Kolmogorov eqn., but without requiring the Markov condition. We present a class of Ito process superficially resembling Markov processes, but with 1-state memory. In finance, such processes would obey the efficient market hypothesis up through the level of pair correlations. These stochastic processes have been mislabeled in recent literature as 'nonlinear Markov processes'. Inspired by Doob and Feller, who pointed out that the ChapmanKolmogorov eqn. is not restricted to Markov processes, we exhibit a Gaussian Ito transition density with 1-state memory in the drift coefficient that satisfies both of Kolmogorov's partial differential eqns. and also the Chapman-Kolmogorov eqn. In addition, we show that three of the examples from McKean's seminal 1966 paper are also nonMarkov Ito processes. Last, we show that the transition density of the generalized Black-Scholes type partial differential eqn. describes a martingale, and satisfies the ChapmanKolmogorov eqn. This leads to the shortest-known proof that the Green function of the Black-Scholes eqn. with variable diffusion coefficient provides the so-called martingale measure of option pricing.

  3. Generated dynamics of Markov and quantum processes

    CERN Document Server

    Janßen, Martin

    2016-01-01

    This book presents Markov and quantum processes as two sides of a coin called generated stochastic processes. It deals with quantum processes as reversible stochastic processes generated by one-step unitary operators, while Markov processes are irreversible stochastic processes generated by one-step stochastic operators. The characteristic feature of quantum processes are oscillations, interference, lots of stationary states in bounded systems and possible asymptotic stationary scattering states in open systems, while the characteristic feature of Markov processes are relaxations to a single stationary state. Quantum processes apply to systems where all variables, that control reversibility, are taken as relevant variables, while Markov processes emerge when some of those variables cannot be followed and are thus irrelevant for the dynamic description. Their absence renders the dynamic irreversible. A further aim is to demonstrate that almost any subdiscipline of theoretical physics can conceptually be put in...

  4. Markov and semi-Markov processes as a failure rate

    Science.gov (United States)

    Grabski, Franciszek

    2016-06-01

    In this paper the reliability function is defined by the stochastic failure rate process with a non negative and right continuous trajectories. Equations for the conditional reliability functions of an object, under assumption that the failure rate is a semi-Markov process with an at most countable state space are derived. A proper theorem is presented. The linear systems of equations for the appropriate Laplace transforms allow to find the reliability functions for the alternating, the Poisson and the Furry-Yule failure rate processes.

  5. Finite Markov processes and their applications

    CERN Document Server

    Iosifescu, Marius

    2007-01-01

    A self-contained treatment of finite Markov chains and processes, this text covers both theory and applications. Author Marius Iosifescu, vice president of the Romanian Academy and director of its Center for Mathematical Statistics, begins with a review of relevant aspects of probability theory and linear algebra. Experienced readers may start with the second chapter, a treatment of fundamental concepts of homogeneous finite Markov chain theory that offers examples of applicable models.The text advances to studies of two basic types of homogeneous finite Markov chains: absorbing and ergodic ch

  6. Markov processes an introduction for physical scientists

    CERN Document Server

    Gillespie, Daniel T

    1991-01-01

    Markov process theory is basically an extension of ordinary calculus to accommodate functions whos time evolutions are not entirely deterministic. It is a subject that is becoming increasingly important for many fields of science. This book develops the single-variable theory of both continuous and jump Markov processes in a way that should appeal especially to physicists and chemists at the senior and graduate level.Key Features* A self-contained, prgamatic exposition of the needed elements of random variable theory* Logically integrated derviations of the Chapman-Kolmogorov e

  7. A Noether Theorem for Markov Processes

    CERN Document Server

    Baez, John C

    2012-01-01

    Noether's theorem links the symmetries of a quantum system with its conserved quantities, and is a cornerstone of quantum mechanics. Here we prove a version of Noether's theorem for Markov processes. In quantum mechanics, an observable commutes with the Hamiltonian if and only if its expected value remains constant in time for every state. For Markov processes that no longer holds, but an observable commutes with the Hamiltonian if and only if both its expected value and standard deviation are constant in time for every state.

  8. One-Counter Markov Decision Processes

    NARCIS (Netherlands)

    Brazdil, T.; Brozek, V.; Etessami, K.; Kucera, A.; Wojtczak, D.K.; Charikar, M.

    2010-01-01

    We study the computational complexity of central analysis problems for One-Counter Markov Decision Processes (OC-MDPs), a class of finitely-presented, countable-state MDPs. OC-MDPs are equivalent to a controlled extension of (discrete-time) Quasi-Birth-Death processes (QBDs), a stochastic model stud

  9. Piecewise deterministic Markov processes : an analytic approach

    NARCIS (Netherlands)

    Alkurdi, Taleb Salameh Odeh

    2013-01-01

    The subject of this thesis, piecewise deterministic Markov processes, an analytic approach, is on the border between analysis and probability theory. Such processes can either be viewed as random perturbations of deterministic dynamical systems in an impulsive fashion, or as a particular kind of

  10. Markov decision processes in artificial intelligence

    CERN Document Server

    Sigaud, Olivier

    2013-01-01

    Markov Decision Processes (MDPs) are a mathematical framework for modeling sequential decision problems under uncertainty as well as Reinforcement Learning problems. Written by experts in the field, this book provides a global view of current research using MDPs in Artificial Intelligence. It starts with an introductory presentation of the fundamental aspects of MDPs (planning in MDPs, Reinforcement Learning, Partially Observable MDPs, Markov games and the use of non-classical criteria). Then it presents more advanced research trends in the domain and gives some concrete examples using illustr

  11. Shift ergodicity for stationary Markov processes

    Institute of Scientific and Technical Information of China (English)

    东金文

    2001-01-01

    In this paper shift ergodicity and related topics are studied for certain stationary processes. We first present a simple proof of the conclusion that every stationary Markov process is a generalized convex combination of stationary ergodic Markov processes. A direct consequence is that a stationary distribution of a Markov process is extremal if and only if the corresponding stationary Markov process is time ergodic and every stationary distribution is a generalized convex combination of such extremal ones. We then consider space ergodicity for spin flip particle systems. We prove space shift ergodicity and mixing for certain extremal invariant measures for a class of spin systems, in which most of the typical models, such as the Voter Models and the Contact Models, are included. As a consequence of these results we see that for such systems, under each of those extremal invariant measures, the space and time means of an observable coincide, an important phenomenon in statistical physics. Our results provide partial answers to certain interesting problems in spin systems.

  12. Renewal characterization of Markov modulated Poisson processes

    Directory of Open Access Journals (Sweden)

    Marcel F. Neuts

    1989-01-01

    Full Text Available A Markov Modulated Poisson Process (MMPP M(t defined on a Markov chain J(t is a pure jump process where jumps of M(t occur according to a Poisson process with intensity λi whenever the Markov chain J(t is in state i. M(t is called strongly renewal (SR if M(t is a renewal process for an arbitrary initial probability vector of J(t with full support on P={i:λi>0}. M(t is called weakly renewal (WR if there exists an initial probability vector of J(t such that the resulting MMPP is a renewal process. The purpose of this paper is to develop general characterization theorems for the class SR and some sufficiency theorems for the class WR in terms of the first passage times of the bivariate Markov chain [J(t,M(t]. Relevance to the lumpability of J(t is also studied.

  13. Bayesian analysis of Markov point processes

    DEFF Research Database (Denmark)

    Berthelsen, Kasper Klitgaard; Møller, Jesper

    2006-01-01

    Recently Møller, Pettitt, Berthelsen and Reeves introduced a new MCMC methodology for drawing samples from a posterior distribution when the likelihood function is only specified up to a normalising constant. We illustrate the method in the setting of Bayesian inference for Markov point processes...

  14. An Explicit Microreversibility Violating Thermodynamic Markov Process

    CERN Document Server

    Lee, Michael J

    2008-01-01

    We explicitly construct a non-microreversible transition matrix for a Markov process and apply it to the standard three-state Potts model. This provides a clear and simple demonstration that the usual micoreversibility property of thermodynamical Monte Carlo algorithms is not strictly necessary from a mathemetical point of view.

  15. Weak Markov Processes as Linear Systems

    CERN Document Server

    Gohm, Rolf

    2012-01-01

    A noncommutative Fornasini-Marchesini system (a multi-variable version of a linear system) can be realized within a weak Markov process (a model for quantum evolution). For a discrete time parameter this is worked out systematically as a theory of representations of structure maps of a system by a weak process. We introduce subprocesses and quotient processes which can be described naturally by a suitable category of weak processes. A corresponding notion of cascade for processes induces a represented cascade of systems. We study the control theoretic notion of observability which turns out to be particularly interesting in connection with a cascade structure. As an application we gain new insights into stationary Markov chains where observability for the system is closely related to asymptotic completeness in the scattering theory of the chain. This motivates a general definition of asymptotic completeness in the category of weak processes.

  16. Shift ergodicity for stationary Markov processes

    Institute of Scientific and Technical Information of China (English)

    CHEN; Jinwen(

    2001-01-01

    [1]Liggett, T. M., Interacting Particle Systems, New York: Springer-Verlag, 1985.[2]Andjel, E. D., Ergodic and mixing properties of equilibrium measures for Markov processes, Trans. of the AMS, 1990, 318:601-614.[3]Franchi, J., Asymptotic windings of ergodic diffusion, Stoch. Processes Appl., 1996, 62: 277-298.[4]Golden, K. , Goldstein, S., Lebowitz, J. L., Nash estimates and the asymptotic behavior of diffusion, Ann. Prob., 1988,16: 1127-1146.[5]Gordin. M. I., Lifsic, B. A. , The central limit theorem for stationary ergodic Markov process, Dokl, Akad. Nauk SSSR,1978, 19: 392-393.[6]Orey, S., Large deviations in ergodic theory, Seminar on Stochastic Processes, 1985, 12: 195-249.[7]Veretenikov, A. Y., On large deviations for ergodic process empirical measures, Adv. Sov. Math., 1992, 12: 125-133.[8]Deuschel, J. D., Stroock, D. W., Large Deviations, San Diego, CA: Academic Press, 1989.[9]Rosenblatt, M., Markov Processes, Structure and Asymptotic Behavior, Berlin: Springer-Verlag, 1971.[10]Liggett, T. M., Stochastic Interacting Systems: Contact, Voter, and Exclusion Processes, Berlin: Springer-Verlag, 1999.[11]Albevrio, S., Kondratiev, Y. G., Rockner, M., Ergodicity of L2-semigroups and extremality of Gibbs states, J. Funct.Anal. , 1997, 144: 293-423.[12]Liverani, C. , Olla, S. , Ergodicity in infinite Hamiltonian systems with conservative noise, Probab. Th. Rel. Fields, 1996,106: 401-445.[13]Varadhan, S. P. S., Large Deviations and Applications, Philadelphia: SIAM, 1984.[14]Chen, J. W. , A variational principle for Markov processes, J. Stat. Phys. , 1999, 96: 1359-1364.

  17. Semigroups, boundary value problems and Markov processes

    CERN Document Server

    Taira, Kazuaki

    2014-01-01

    A careful and accessible exposition of functional analytic methods in stochastic analysis is provided in this book. It focuses on the interrelationship between three subjects in analysis: Markov processes, semi groups and elliptic boundary value problems. The author studies a general class of elliptic boundary value problems for second-order, Waldenfels integro-differential operators in partial differential equations and proves that this class of elliptic boundary value problems provides a general class of Feller semigroups in functional analysis. As an application, the author constructs a general class of Markov processes in probability in which a Markovian particle moves both by jumps and continuously in the state space until it 'dies' at the time when it reaches the set where the particle is definitely absorbed. Augmenting the 1st edition published in 2004, this edition includes four new chapters and eight re-worked and expanded chapters. It is amply illustrated and all chapters are rounded off with Notes ...

  18. Modelling the Heterogeneous Markov Attrition Process .

    Directory of Open Access Journals (Sweden)

    Jau Yeu Menq

    1993-01-01

    Full Text Available A model for heterogeneous dynamics combat as a continuos-time Markov process has been studied, and on account of the special form of its infinitesimal generator, recursive algorithms are derived to compute the important characteristics of the combat, such as the combat time distribution, expected value and variance, and the probability of winning and expected survivors. Numerical results are also presented. This approach can also be used to consider initial contact forces of both sides as random variables.

  19. Metastability for Markov processes with detailed balance.

    Science.gov (United States)

    Larralde, Hernán; Leyvraz, François

    2005-04-29

    We present a definition for metastable states applicable to arbitrary finite state Markov processes satisfying detailed balance. In particular, we identify a crucial condition that distinguishes metastable states from other slow decaying modes and which allows us to show that our definition has several desirable properties similar to those postulated in the restricted ensemble approach. The intuitive physical meaning of this condition is simply that the total equilibrium probability of finding the system in the metastable state is negligible.

  20. Metrics for Finite Markov Decision Processes

    OpenAIRE

    Ferns, Norman; Panangaden, Prakash; Precup, Doina

    2012-01-01

    We present metrics for measuring the similarity of states in a finite Markov decision process (MDP). The formulation of our metrics is based on the notion of bisimulation for MDPs, with an aim towards solving discounted infinite horizon reinforcement learning tasks. Such metrics can be used to aggregate states, as well as to better structure other value function approximators (e.g., memory-based or nearest-neighbor approximators). We provide bounds that relate our metric distances to the opti...

  1. Markov Chain Approximations to Singular Stable-like Processes

    OpenAIRE

    2012-01-01

    We consider the Markov chain approximations for singular stable-like processes. First we obtain properties of some Markov chains. Then we construct the approximating Markov chains and give a necessary condition for weak convergence of these chains to singular stable-like processes.

  2. Boundary value problems and Markov processes

    CERN Document Server

    Taira, Kazuaki

    2009-01-01

    This volume is devoted to a thorough and accessible exposition on the functional analytic approach to the problem of construction of Markov processes with Ventcel' boundary conditions in probability theory. Analytically, a Markovian particle in a domain of Euclidean space is governed by an integro-differential operator, called a Waldenfels operator, in the interior of the domain, and it obeys a boundary condition, called the Ventcel' boundary condition, on the boundary of the domain. Probabilistically, a Markovian particle moves both by jumps and continuously in the state space and it obeys the Ventcel' boundary condition, which consists of six terms corresponding to the diffusion along the boundary, the absorption phenomenon, the reflection phenomenon, the sticking (or viscosity) phenomenon, the jump phenomenon on the boundary, and the inward jump phenomenon from the boundary. In particular, second-order elliptic differential operators are called diffusion operators and describe analytically strong Markov pr...

  3. Continuity Properties of Distances for Markov Processes

    DEFF Research Database (Denmark)

    Jaeger, Manfred; Mao, Hua; Larsen, Kim Guldstrand

    2014-01-01

    In this paper we investigate distance functions on finite state Markov processes that measure the behavioural similarity of non-bisimilar processes. We consider both probabilistic bisimilarity metrics, and trace-based distances derived from standard Lp and Kullback-Leibler distances. Two desirable...... continuity properties for such distances are identified. We then establish a number of results that show that these two properties are in conflict, and not simultaneously fulfilled by any of our candidate natural distance functions. An impossibility result is derived that explains to some extent...

  4. Hybrid Discrete-Continuous Markov Decision Processes

    Science.gov (United States)

    Feng, Zhengzhu; Dearden, Richard; Meuleau, Nicholas; Washington, Rich

    2003-01-01

    This paper proposes a Markov decision process (MDP) model that features both discrete and continuous state variables. We extend previous work by Boyan and Littman on the mono-dimensional time-dependent MDP to multiple dimensions. We present the principle of lazy discretization, and piecewise constant and linear approximations of the model. Having to deal with several continuous dimensions raises several new problems that require new solutions. In the (piecewise) linear case, we use techniques from partially- observable MDPs (POMDPS) to represent value functions as sets of linear functions attached to different partitions of the state space.

  5. Synchronizing Objectives for Markov Decision Processes

    CERN Document Server

    Doyen, Laurent; Shirmohammadi, Mahsa; 10.4204/EPTCS.50.5

    2011-01-01

    We introduce synchronizing objectives for Markov decision processes (MDP). Intuitively, a synchronizing objective requires that eventually, at every step there is a state which concentrates almost all the probability mass. In particular, it implies that the probabilistic system behaves in the long run like a deterministic system: eventually, the current state of the MDP can be identified with almost certainty. We study the problem of deciding the existence of a strategy to enforce a synchronizing objective in MDPs. We show that the problem is decidable for general strategies, as well as for blind strategies where the player cannot observe the current state of the MDP. We also show that pure strategies are sufficient, but memory may be necessary.

  6. Transition-Independent Decentralized Markov Decision Processes

    Science.gov (United States)

    Becker, Raphen; Silberstein, Shlomo; Lesser, Victor; Goldman, Claudia V.; Morris, Robert (Technical Monitor)

    2003-01-01

    There has been substantial progress with formal models for sequential decision making by individual agents using the Markov decision process (MDP). However, similar treatment of multi-agent systems is lacking. A recent complexity result, showing that solving decentralized MDPs is NEXP-hard, provides a partial explanation. To overcome this complexity barrier, we identify a general class of transition-independent decentralized MDPs that is widely applicable. The class consists of independent collaborating agents that are tied up by a global reward function that depends on both of their histories. We present a novel algorithm for solving this class of problems and examine its properties. The result is the first effective technique to solve optimally a class of decentralized MDPs. This lays the foundation for further work in this area on both exact and approximate solutions.

  7. Boundary value problems and Markov processes

    CERN Document Server

    Taira, Kazuaki

    1991-01-01

    Focussing on the interrelations of the subjects of Markov processes, analytic semigroups and elliptic boundary value problems, this monograph provides a careful and accessible exposition of functional methods in stochastic analysis. The author studies a class of boundary value problems for second-order elliptic differential operators which includes as particular cases the Dirichlet and Neumann problems, and proves that this class of boundary value problems provides a new example of analytic semigroups both in the Lp topology and in the topology of uniform convergence. As an application, one can construct analytic semigroups corresponding to the diffusion phenomenon of a Markovian particle moving continuously in the state space until it "dies", at which time it reaches the set where the absorption phenomenon occurs. A class of initial-boundary value problems for semilinear parabolic differential equations is also considered. This monograph will appeal to both advanced students and researchers as an introductio...

  8. Safe Exploration in Markov Decision Processes

    CERN Document Server

    Moldovan, Teodor Mihai

    2012-01-01

    In environments with uncertain dynamics exploration is necessary to learn how to perform well. Existing reinforcement learning algorithms provide strong exploration guarantees, but they tend to rely on an ergodicity assumption. The essence of ergodicity is that any state is eventually reachable from any other state by following a suitable policy. This assumption allows for exploration algorithms that operate by simply favoring states that have rarely been visited before. For most physical systems this assumption is impractical as the systems would break before any reasonable exploration has taken place, i.e., most physical systems don't satisfy the ergodicity assumption. In this paper we address the need for safe exploration methods in Markov decision processes. We first propose a general formulation of safety through ergodicity. We show that imposing safety by restricting attention to the resulting set of guaranteed safe policies is NP-hard. We then present an efficient algorithm for guaranteed safe, but pot...

  9. Wiener-Hopf factorizations for a multidimensional Markov additive process and their applications to reflected processes

    Directory of Open Access Journals (Sweden)

    Masakiyo Miyazawa

    2012-01-01

    Full Text Available We extend the framework of Neuts' matrix analytic approach to a reflected process generated by a discrete time multidimensional Markov additive process. This Markov additive process has a general background state space and a real vector valued additive component, and generates a multidimensional reflected process. Our major interest is to derive a closed form formula for the stationary distribution of this reflected process. To this end, we introduce a real valued level, and derive new versions of the Wiener-Hopf factorization for the Markov additive process with the multidimensional additive component. In particular, itis represented by moment generating functions, and we consider the domain for it to be valid.Our framework is general enough to include multi-server queues and/or queueing networks as well as non-linear time series which are currently popular in financial and actuarial mathematics. Our results yield structural results for such models. As an illustration, we apply our results to extend existing results on the tail behavior of reflected processes.A major theme of this work is to connect recent work on matrix analytic methods to classical probabilistic studies on Markov additive processes. Indeed, using purely probabilistic methods such as censoring, duality, level crossing and time-reversal (which are known in the matrix analytic methods community but date back to Arjas & Speed [2] and Pitman [29], we extend and unify existing results in both areas.

  10. Transformation of state space for two-parameter Markov processes

    Institute of Scientific and Technical Information of China (English)

    周健伟

    1996-01-01

    Let X=(X) be a two-parameter *-Markov process with a transition function (p1, p2, p), where X, takes values in the state space (Er,), T=[0,)2. For each r T, let f, be a measurable transformation of (E,) into the state space (E’r, ). Set Y,=f,(X,), r T. A sufficient condition is given for the process Y=(Yr) still to be a two-parameter *-Markov process with a transition function in terms of transition function (p1, p2, p) and fr. For *-Markov families of two-parameter processes with a transition function, a similar problem is also discussed.

  11. Relativized hierarchical decomposition of Markov decision processes.

    Science.gov (United States)

    Ravindran, B

    2013-01-01

    Reinforcement Learning (RL) is a popular paradigm for sequential decision making under uncertainty. A typical RL algorithm operates with only limited knowledge of the environment and with limited feedback on the quality of the decisions. To operate effectively in complex environments, learning agents require the ability to form useful abstractions, that is, the ability to selectively ignore irrelevant details. It is difficult to derive a single representation that is useful for a large problem setting. In this chapter, we describe a hierarchical RL framework that incorporates an algebraic framework for modeling task-specific abstraction. The basic notion that we will explore is that of a homomorphism of a Markov Decision Process (MDP). We mention various extensions of the basic MDP homomorphism framework in order to accommodate different commonly understood notions of abstraction, namely, aspects of selective attention. Parts of the work described in this chapter have been reported earlier in several papers (Narayanmurthy and Ravindran, 2007, 2008; Ravindran and Barto, 2002, 2003a,b; Ravindran et al., 2007).

  12. Markov processes from K. Ito's perspective (AM-155)

    CERN Document Server

    Stroock, Daniel W

    2003-01-01

    Kiyosi Itô''s greatest contribution to probability theory may be his introduction of stochastic differential equations to explain the Kolmogorov-Feller theory of Markov processes. Starting with the geometric ideas that guided him, this book gives an account of Itô''s program. The modern theory of Markov processes was initiated by A. N. Kolmogorov. However, Kolmogorov''s approach was too analytic to reveal the probabilistic foundations on which it rests. In particular, it hides the central role played by the simplest Markov processes: those with independent, identically distributed incremen

  13. Continuous-time Markov decision processes theory and applications

    CERN Document Server

    Guo, Xianping

    2009-01-01

    This volume provides the first book entirely devoted to recent developments on the theory and applications of continuous-time Markov decision processes (MDPs). The MDPs presented here include most of the cases that arise in applications.

  14. RANDOM TIMES TRANSFORMATION OF PROCESSES WITH MARKOV SKELETON%Markov 骨架过程的随机时变换

    Institute of Scientific and Technical Information of China (English)

    刘万荣; 刘再明; 侯振挺

    2000-01-01

    In this paper, random time transformations of processes with Markov skeleton are discussed. A class of random time transformations that transform a process with Markov skeleton into a process with Markov skeleton is given.%讨论了 Markov 骨架过程的随机时变换,给出了一类变换 Markov 骨架过程为Markov 骨架过程的随机时变换.

  15. Markov

    Directory of Open Access Journals (Sweden)

    Carlos Alejandro De Luna Ortega

    2006-01-01

    Full Text Available En este artículo se aborda el diseño de un reconocedor de voz, con el idioma español mexicano, del estado de Aguascalientes, de palabras aisladas, con dependencia del hablante y vocabulario pequeño, empleando Redes Neuronales Artificiales (ANN por sus siglas en inglés, Alineamiento Dinámico del Tiempo (DTW por sus siglas en inglés y Modelos Ocultos de Markov (HMM por sus siglas en inglés para la realización del algoritmo de reconocimiento.

  16. Poisson point processes and their application to Markov processes

    CERN Document Server

    Itô, Kiyosi

    2015-01-01

    An extension problem (often called a boundary problem) of Markov processes has been studied, particularly in the case of one-dimensional diffusion processes, by W. Feller, K. Itô, and H. P. McKean, among others. In this book, Itô discussed a case of a general Markov process with state space S and a specified point a ∈ S called a boundary. The problem is to obtain all possible recurrent extensions of a given minimal process (i.e., the process on S \\ {a} which is absorbed on reaching the boundary a). The study in this lecture is restricted to a simpler case of the boundary a being a discontinuous entrance point, leaving a more general case of a continuous entrance point to future works. He established a one-to-one correspondence between a recurrent extension and a pair of a positive measure k(db) on S \\ {a} (called the jumping-in measure and a non-negative number m< (called the stagnancy rate). The necessary and sufficient conditions for a pair k, m was obtained so that the correspondence is precisely de...

  17. Gelation of a Reversible Markov Process of Polymerization

    Institute of Scientific and Technical Information of China (English)

    Dong Han; Yian-lin Han

    2003-01-01

    In this paper a reversible Markov process as a chemical polymers reaction of two types of monomers is defined. By analyzing the partition functions of the process we obtain three different distributions of the average molecular weight, depending on the value of strength of the fragmentation reaction, and prove that a gelation of the process will occur in the thermodynamic limit.

  18. Nash Inequalities for Markov Processes in Dimension One

    Institute of Scientific and Technical Information of China (English)

    MAO Yong Hua

    2002-01-01

    In this paper, we give characterizations of Nash inequalities for birth-death process anddiffusion process on the line. As a by-product, we prove that for these processes, transience impliesthat the semigroups P(t) decay as‖P(t)‖1→∞≤ Ct-1.Sufficient conditions for general Markov chains are also obtained.

  19. Construction of Markov process with Markov resolvent on Lp(E,m)

    Institute of Scientific and Technical Information of China (English)

    董昭

    1997-01-01

    Lei E be a Hausdorff topological space and let A(E) be its Borel σ-field.Let m be a σ-finitc measure on (E,A(E)).A necessary and sufficient condition for a Markov resolvent on Lp(E,m ) to be associated with an m-tight m-special standard process (with state space E) is given.Furthermore some new examples which do not belong to the framework of Dirichlet space are also given.

  20. Efficient maximum likelihood parameterization of continuous-time Markov processes

    CERN Document Server

    McGibbon, Robert T

    2015-01-01

    Continuous-time Markov processes over finite state-spaces are widely used to model dynamical processes in many fields of natural and social science. Here, we introduce an maximum likelihood estimator for constructing such models from data observed at a finite time interval. This estimator is drastically more efficient than prior approaches, enables the calculation of deterministic confidence intervals in all model parameters, and can easily enforce important physical constraints on the models such as detailed balance. We demonstrate and discuss the advantages of these models over existing discrete-time Markov models for the analysis of molecular dynamics simulations.

  1. Acceleration of association‐rule based markov decision processes

    Directory of Open Access Journals (Sweden)

    Ma. de G. García‐Hernández

    2009-12-01

    Full Text Available In this paper, we present a new approach for the estimation of Markov decision processes based on efficient association rulemining techniques such as Apriori. For the fastest solution of the resulting association‐rule based Markov decision process,several accelerating procedures such as asynchronous updates and prioritization using a static ordering have been applied. Anew criterion for state reordering in decreasing order of maximum reward is also compared with a modified topologicalreordering algorithm. Experimental results obtained on a finite state and action‐space stochastic shortest path problemdemonstrate the feasibility of the new approach.

  2. Stochastic model of milk homogenization process using Markov's chain

    OpenAIRE

    A. A. Khvostov; R. S. Sumina; G. I. Kotov; Ivanov, A. V.

    2016-01-01

    The process of development of a mathematical model of the process of homogenization of dairy products is considered in the work. The theory of Markov's chains was used in the development of the mathematical model, Markov's chain with discrete states and continuous parameter for which the homogenisation pressure is taken, being the basis for the model structure. Machine realization of the model is implemented in the medium of structural modeling MathWorks Simulink™. Identification of the model...

  3. Markov chain Monte Carlo inference for Markov jump processes via the linear noise approximation.

    Science.gov (United States)

    Stathopoulos, Vassilios; Girolami, Mark A

    2013-02-13

    Bayesian analysis for Markov jump processes (MJPs) is a non-trivial and challenging problem. Although exact inference is theoretically possible, it is computationally demanding, thus its applicability is limited to a small class of problems. In this paper, we describe the application of Riemann manifold Markov chain Monte Carlo (MCMC) methods using an approximation to the likelihood of the MJP that is valid when the system modelled is near its thermodynamic limit. The proposed approach is both statistically and computationally efficient whereas the convergence rate and mixing of the chains allow for fast MCMC inference. The methodology is evaluated using numerical simulations on two problems from chemical kinetics and one from systems biology.

  4. The maximal process of nonlinear shot noise

    Science.gov (United States)

    Eliazar, Iddo; Klafter, Joseph

    2009-05-01

    In the nonlinear shot noise system-model shots’ statistics are governed by general Poisson processes, and shots’ decay-dynamics are governed by general nonlinear differential equations. In this research we consider a nonlinear shot noise system and explore the process tracking, along time, the system’s maximal shot magnitude. This ‘maximal process’ is a stationary Markov process following a decay-surge evolution; it is highly robust, and it is capable of displaying both a wide spectrum of statistical behaviors and a rich variety of random decay-surge sample-path trajectories. A comprehensive analysis of the maximal process is conducted, including its Markovian structure, its decay-surge structure, and its correlation structure. All results are obtained analytically and in closed-form.

  5. ON SHARP MARKOV PROPERTY OF TWO-PARAMETER MARKOV PROCESSES%两参数Markov过程的Sharp Markov性

    Institute of Scientific and Technical Information of China (English)

    陈雄

    2011-01-01

    It is proven that for two-parameter Markov processes, only when they are the finite or countable unions of rectangles, will they show Sharp Markov property.%证明了两参数Markov过程仅在正位矩形的有限并或可列并区域上才具有Sharp Markov性.

  6. SINGULARLY PERTURBED MARKOV DECISION PROCESSES WITH INCLUSION OF TRANSIENT STATES

    Institute of Scientific and Technical Information of China (English)

    R. H. Liu; Q. Zhang; G. Yin

    2001-01-01

    This paper is concerned with the continuous-time Markov decision processes (MDP) having weak and strong interactions. Using a hierarchical approach, the state space of the underlying Markov chain can be decomposed into several groups of recurrent states and a group of transient states resulting in a singularly perturbed MDP formulation.Instead of solving the original problem directly, a limit problem that is much simpler to handle is derived. On the basis of the optimal solution of the limit problem, nearly optimal decisions are constructed for the original problem. The asymptotic optimality of the constructed control is obtained; the rate of convergence is ascertained.

  7. Hidden Markov processes theory and applications to biology

    CERN Document Server

    Vidyasagar, M

    2014-01-01

    This book explores important aspects of Markov and hidden Markov processes and the applications of these ideas to various problems in computational biology. The book starts from first principles, so that no previous knowledge of probability is necessary. However, the work is rigorous and mathematical, making it useful to engineers and mathematicians, even those not interested in biological applications. A range of exercises is provided, including drills to familiarize the reader with concepts and more advanced problems that require deep thinking about the theory. Biological applications are t

  8. a Markov-Process Inspired CA Model of Highway Traffic

    Science.gov (United States)

    Wang, Fa; Li, Li; Hu, Jian-Ming; Ji, Yan; Ma, Rui; Jiang, Rui

    To provide a more accurate description of the driving behaviors especially in car-following, namely a Markov-Gap cellular automata model is proposed in this paper. It views the variation of the gap between two consequent vehicles as a Markov process whose stationary distribution corresponds to the observed gap distribution. This new model provides a microscopic simulation explanation for the governing interaction forces (potentials) between the queuing vehicles, which cannot be directly measurable for traffic flow applications. The agreement between empirical observations and simulation results suggests the soundness of this new approach.

  9. THE FEYNMAN-KAC FORMULA FOR SYMMETRIC MARKOV PROCESSES

    Institute of Scientific and Technical Information of China (English)

    YINGJIANGANG

    1997-01-01

    Let X be an m-symmetric Markov process and M a multiplicative functional of X such that the M-subprocess of X is also m-symmetric. The author characterizes the Dirichlet form associated with the subprocess in terms of that associated with X and the bivariate Revuz measure of M.

  10. Markov Skeleton Processes and Applications to Queueing Systems

    Institute of Scientific and Technical Information of China (English)

    Zhen-ting Hou

    2002-01-01

    In this paper, we apply the backward equations of Markov skeleton processes to queueing systems.The transient distribution of the waiting time of a GI/G/1 queueing system, the transient distribution of the length of a GI/G/N queueing system and the transient distribution of the length of queueing networks are obtained.

  11. Elements of the theory of Markov processes and their applications

    CERN Document Server

    Bharucha-Reid, A T

    2010-01-01

    This graduate-level text and reference in probability, with numerous applications to several fields of science, presents nonmeasure-theoretic introduction to theory of Markov processes. The work also covers mathematical models based on the theory, employed in various applied fields. Prerequisites are a knowledge of elementary probability theory, mathematical statistics, and analysis. Appendixes. Bibliographies. 1960 edition.

  12. Non-parametric Bayesian inference for inhomogeneous Markov point processes

    DEFF Research Database (Denmark)

    Berthelsen, Kasper Klitgaard; Møller, Jesper

    With reference to a specific data set, we consider how to perform a flexible non-parametric Bayesian analysis of an inhomogeneous point pattern modelled by a Markov point process, with a location dependent first order term and pairwise interaction only. A priori we assume that the first order term...

  13. Synthesis for PCTL in Parametric Markov Decision Processes

    DEFF Research Database (Denmark)

    Hahn, Ernst Moritz; Han, Tingting; Zhang, Lijun

    2011-01-01

    In parametric Markov decision processes (PMDPs), transition probabilities are not fixed, but are given as functions over a set of parameters. A PMDP denotes a family of concrete MDPs. This paper studies the synthesis problem for PCTL in PMDPs: Given a specification Φ in PCTL, we synthesise the pa...

  14. Safety Verification of Piecewise-Deterministic Markov Processes

    DEFF Research Database (Denmark)

    Wisniewski, Rafael; Sloth, Christoffer; Bujorianu, Manuela

    2016-01-01

    We consider the safety problem of piecewise-deterministic Markov processes (PDMP). These are systems that have deterministic dynamics and stochastic jumps, where both the time and the destination of the jumps are stochastic. Specifically, we solve a p-safety problem, where we identify the set...

  15. Markov Limid processes for representing and solving renewal problems

    DEFF Research Database (Denmark)

    Jørgensen, Erik; Kristensen, Anders R.; Nilsson, Dennis

    2014-01-01

    In this paper a new tool for simultaneous optimisation of decisions on multiple time scales is presented. The tool combines the dynamic properties of Markov decision processes with the flexible and compact state space representation of LImited Memory Influence Diagrams (Limids). A temporal version...

  16. Bearing Degradation Process Prediction Based on the Support Vector Machine and Markov Model

    Directory of Open Access Journals (Sweden)

    Shaojiang Dong

    2014-01-01

    Full Text Available Predicting the degradation process of bearings before they reach the failure threshold is extremely important in industry. This paper proposed a novel method based on the support vector machine (SVM and the Markov model to achieve this goal. Firstly, the features are extracted by time and time-frequency domain methods. However, the extracted original features are still with high dimensional and include superfluous information, and the nonlinear multifeatures fusion technique LTSA is used to merge the features and reduces the dimension. Then, based on the extracted features, the SVM model is used to predict the bearings degradation process, and the CAO method is used to determine the embedding dimension of the SVM model. After the bearing degradation process is predicted by SVM model, the Markov model is used to improve the prediction accuracy. The proposed method was validated by two bearing run-to-failure experiments, and the results proved the effectiveness of the methodology.

  17. HILBERTIAN INVARIANCE PRINCIPLE FOR EMPIRICAL PROCESS ASSOCIATED WITH A MARKOV PROCESS

    Institute of Scientific and Technical Information of China (English)

    蒋义文; 吴黎明

    2003-01-01

    The authors establish the Hilbertian invariance principle for the empirical process of astationary Markov process, by extending the forward-backward martingale decomposition ofLyons-Meyer-Zheng to the Hilbert space valued additive functionals associated with generalnon-reversible Markov processes.

  18. The construction of Markov processes in random environments and the equivalence theorems

    Institute of Scientific and Technical Information of China (English)

    2004-01-01

    In sec.1, we introduce several basic concepts such as random transition function, p-m process and Markov process in random environment and give some examples to construct a random transition function from a non-homogeneous density function. In sec.2, we construct the Markov process in random enviromment and skew product Markov process by p - m process and investigate the properties of Markov process in random environment and the original process and environment process and skew product process. In sec. 3, we give several equivalence theorems on Markov process in random environment.

  19. Multiresolution Hilbert Approach to Multidimensional Gauss-Markov Processes

    Directory of Open Access Journals (Sweden)

    Thibaud Taillefumier

    2011-01-01

    Full Text Available The study of the multidimensional stochastic processes involves complex computations in intricate functional spaces. In particular, the diffusion processes, which include the practically important Gauss-Markov processes, are ordinarily defined through the theory of stochastic integration. Here, inspired by the Lévy-Ciesielski construction of the Wiener process, we propose an alternative representation of multidimensional Gauss-Markov processes as expansions on well-chosen Schauder bases, with independent random coefficients of normal law with zero mean and unit variance. We thereby offer a natural multiresolution description of the Gauss-Markov processes as limits of finite-dimensional partial sums of the expansion, that are strongly almost-surely convergent. Moreover, such finite-dimensional random processes constitute an optimal approximation of the process, in the sense of minimizing the associated Dirichlet energy under interpolating constraints. This approach allows for a simpler treatment of problems in many applied and theoretical fields, and we provide a short overview of applications we are currently developing.

  20. MARKOV: A methodology for the solution of infinite time horizon MARKOV decision processes

    Science.gov (United States)

    Williams, B.K.

    1988-01-01

    Algorithms are described for determining optimal policies for finite state, finite action, infinite discrete time horizon Markov decision processes. Both value-improvement and policy-improvement techniques are used in the algorithms. Computing procedures are also described. The algorithms are appropriate for processes that are either finite or infinite, deterministic or stochastic, discounted or undiscounted, in any meaningful combination of these features. Computing procedures are described in terms of initial data processing, bound improvements, process reduction, and testing and solution. Application of the methodology is illustrated with an example involving natural resource management. Management implications of certain hypothesized relationships between mallard survival and harvest rates are addressed by applying the optimality procedures to mallard population models.

  1. Active Learning of Markov Decision Processes for System Verification

    DEFF Research Database (Denmark)

    Chen, Yingke; Nielsen, Thomas Dyhre

    2012-01-01

    of input/output observations. While alleviating the problem of manually constructing a system model, the collection/generation of observed system behaviors can also prove demanding. Consequently we seek to minimize the amount of data required. In this paper we propose an algorithm for learning...... deterministic Markov decision processes from data by actively guiding the selection of input actions. The algorithm is empirically analyzed by learning system models of slot machines, and it is demonstrated that the proposed active learning procedure can significantly reduce the amount of data required...... demanding process, and this shortcoming has motivated the development of algorithms for automatically learning system models from observed system behaviors. Recently, algorithms have been proposed for learning Markov decision process representations of reactive systems based on alternating sequences...

  2. Optimal control for unknown discrete-time nonlinear Markov jump systems using adaptive dynamic programming.

    Science.gov (United States)

    Zhong, Xiangnan; He, Haibo; Zhang, Huaguang; Wang, Zhanshan

    2014-12-01

    In this paper, we develop and analyze an optimal control method for a class of discrete-time nonlinear Markov jump systems (MJSs) with unknown system dynamics. Specifically, an identifier is established for the unknown systems to approximate system states, and an optimal control approach for nonlinear MJSs is developed to solve the Hamilton-Jacobi-Bellman equation based on the adaptive dynamic programming technique. We also develop detailed stability analysis of the control approach, including the convergence of the performance index function for nonlinear MJSs and the existence of the corresponding admissible control. Neural network techniques are used to approximate the proposed performance index function and the control law. To demonstrate the effectiveness of our approach, three simulation studies, one linear case, one nonlinear case, and one single link robot arm case, are used to validate the performance of the proposed optimal control method.

  3. Bisimulation on Markov Processes over Arbitrary Measurable Spaces

    DEFF Research Database (Denmark)

    Bacci, Giorgio; Bacci, Giovanni; Larsen, Kim Guldstrand

    2014-01-01

    We introduce a notion of bisimulation on labelled Markov Processes over generic measurable spaces in terms of arbitrary binary relations. Our notion of bisimulation is proven to coincide with the coalgebraic definition of Aczel and Mendler in terms of the Giry functor, which associates with a mea......We introduce a notion of bisimulation on labelled Markov Processes over generic measurable spaces in terms of arbitrary binary relations. Our notion of bisimulation is proven to coincide with the coalgebraic definition of Aczel and Mendler in terms of the Giry functor, which associates......)category of cocongruences, which gives new insights about the real categorical nature of their results. As a corollary, we obtain sufficient conditions under which state and event bisimilarity coincide....

  4. Symbolic Heuristic Search for Factored Markov Decision Processes

    Science.gov (United States)

    Morris, Robert (Technical Monitor); Feng, Zheng-Zhu; Hansen, Eric A.

    2003-01-01

    We describe a planning algorithm that integrates two approaches to solving Markov decision processes with large state spaces. State abstraction is used to avoid evaluating states individually. Forward search from a start state, guided by an admissible heuristic, is used to avoid evaluating all states. We combine these two approaches in a novel way that exploits symbolic model-checking techniques and demonstrates their usefulness for decision-theoretic planning.

  5. First passage times for Markov renewal processes and applications

    Institute of Scientific and Technical Information of China (English)

    2000-01-01

    This paper proposes a uniformly convergent algorithm for the joint transform of the first passage time and the first passage number of steps for general Markov renewal processes with any initial state probability vector. The uniformly convergent algorithm with arbitrarily prescribed error can be efficiently applied to compute busy periods, busy cycles, waiting times, sojourn times, and relevant indices of various generic queueing systems and queueing networks. This paper also conducts a numerical experiment to implement the proposed algorithm.

  6. Experimental optimization of a real time fed-batch fermentation process using Markov decision process.

    Science.gov (United States)

    Saucedo, V M; Karim, M N

    1997-07-20

    This article describes a methodology that implements a Markov decision process (MDP) optimization technique in a real time fed-batch experiment. Biological systems can be better modeled under the stochastic framework and MDP is shown to be a suitable technique for their optimization. A nonlinear input/output model is used to calculate the probability transitions. All elements of the MDP are identified according to physical parameters. Finally, this study compares the results obtained when optimizing ethanol production using the infinite horizon problem, with total expected discount policy, to previous experimental results aimed at optimizing ethanol production using a recombinant Escherichia coli fed-batch cultivation. (c) 1997 John Wiley & Sons, Inc. Biotechnol Bioeng 55: 317-327, 1997.

  7. Markov decision processes in natural resources management: observability and uncertainty

    Science.gov (United States)

    Williams, Byron K.

    2015-01-01

    The breadth and complexity of stochastic decision processes in natural resources presents a challenge to analysts who need to understand and use these approaches. The objective of this paper is to describe a class of decision processes that are germane to natural resources conservation and management, namely Markov decision processes, and to discuss applications and computing algorithms under different conditions of observability and uncertainty. A number of important similarities are developed in the framing and evaluation of different decision processes, which can be useful in their applications in natural resources management. The challenges attendant to partial observability are highlighted, and possible approaches for dealing with it are discussed.

  8. Semi adiabatic theory of seasonal Markov processes

    Energy Technology Data Exchange (ETDEWEB)

    Talkner, P. [Paul Scherrer Inst. (PSI), Villigen (Switzerland)

    1999-08-01

    The dynamics of many natural and technical systems are essentially influenced by a periodic forcing. Analytic solutions of the equations of motion for periodically driven systems are generally not known. Simulations, numerical solutions or in some limiting cases approximate analytic solutions represent the known approaches to study the dynamics of such systems. Besides the regime of weak periodic forces where linear response theory works, the limit of a slow driving force can often be treated analytically using an adiabatic approximation. For this approximation to hold all intrinsic processes must be fast on the time-scale of a period of the external driving force. We developed a perturbation theory for periodically driven Markovian systems that covers the adiabatic regime but also works if the system has a single slow mode that may even be slower than the driving force. We call it the semi adiabatic approximation. Some results of this approximation for a system exhibiting stochastic resonance which usually takes place within the semi adiabatic regime are indicated. (author) 1 fig., 8 refs.

  9. Multivariate Semi-Markov Process for Counterparty Credit Risk

    CERN Document Server

    D'Amico, Guglielmo; Salvi, Giovanni

    2011-01-01

    In this work we define a multivariate semi-Markov process. We derive an explicit expression for the transition probability of this multivariate semi-Markov process in the discrete time case. We apply this multivariate model to the study of the counterparty credit risk, with regard to correlation in a CDS contract. The financial crisis has stressed the importance of the study of the correlation in the financial market. In this regard, the study of the risk of default of the counterparty in any financial contract has become crucial in the credit risk. Many works has been done to trying to describe the counterparty risk in a CDS contract, but all this work are based on the Markovian approach to risk. In the our opinion this kind of model are too restrictive, because they require that the distribuction function of the waiting times has to be exponential or geometric, for discrete time. In the our model, we describe the evolution of credit rating of the financial subjects like a multivariate semi-Markov model, so ...

  10. Markov chains and decision processes for engineers and managers

    CERN Document Server

    Sheskin, Theodore J

    2010-01-01

    Markov Chain Structure and ModelsHistorical NoteStates and TransitionsModel of the WeatherRandom WalksEstimating Transition ProbabilitiesMultiple-Step Transition ProbabilitiesState Probabilities after Multiple StepsClassification of StatesMarkov Chain StructureMarkov Chain ModelsProblemsReferencesRegular Markov ChainsSteady State ProbabilitiesFirst Passage to a Target StateProblemsReferencesReducible Markov ChainsCanonical Form of the Transition MatrixTh

  11. Partially observable Markov decision processes for risk-based screening

    Science.gov (United States)

    Mrozack, Alex; Liao, Xuejun; Skatter, Sondre; Carin, Lawrence

    2016-05-01

    A long-term goal for checked baggage screening in airports has been to include passenger information, or at least a predetermined passenger risk level, in the screening process. One method for including that information could be treating the checked baggage screening process as a system-of-systems. This would allow for an optimized policy builder, such as one trained using the methodology of partially observable Markov decision processes (POMDP), to navigate the different sensors available for screening. In this paper we describe the necessary steps to tailor a POMDP for baggage screening, as well as results of simulations for specific screening scenarios.

  12. A Separated Approach to Control of Markov Jump Nonlinear Systems With General Transition Probabilities.

    Science.gov (United States)

    Shen, Mouquan; Park, Ju H; Ye, Dan

    2016-09-01

    This paper is devoted to the control of Markov jump nonlinear systems with general transition probabilities (TPs) allowed to be known, uncertain, and unknown. With the help of the S-procedure to dispose the system nonlinearities and the TP property to eliminate the coupling between unknown TP and Lyapunov variable, an extended bounded real lemma for the considered system to be stochastically stable with the prescribed H∞ performance is established in the framework of linear matrix inequalities. To handle the nonlinearity incurred by uncertain TP for controller synthesis, a separated method is proposed to decouple the interconnection between Lyapunov variables and controller gains. A numerical example is given to show the effectiveness of the proposed method.

  13. A new formalism that combines advantages of fault-trees and Markov models: Boolean logic driven Markov processes

    Energy Technology Data Exchange (ETDEWEB)

    Bouissou, Marc; Bon, Jean-Louis

    2003-11-01

    This paper introduces a modeling formalism that enables the analyst to combine concepts inherited from fault trees and Markov models in a new way. We call this formalism Boolean logic Driven Markov Processes (BDMP). It has two advantages over conventional models used in dependability assessment: it allows the definition of complex dynamic models while remaining nearly as readable and easy to build as fault-trees, and it offers interesting mathematical properties, which enable an efficient processing for BDMP that are equivalent to Markov processes with huge state spaces. We give a mathematical definition of BDMP, the demonstration of their properties, and several examples to illustrate how powerful and easy to use they are. From a mathematical point of view, a BDMP is nothing more than a certain way to define a global Markov process, as the result of several elementary processes which can interact in a given manner. An extreme case is when the processes are independent. Then we simply have a fault-tree, the leaves of which are associated to independent Markov processes.

  14. sl(2) Operators and Markov Processes on Branching Graphs

    CERN Document Server

    Petrov, Leonid

    2011-01-01

    We present a unified approach to various examples of Markov dynamics on partitions studied by Borodin, Olshanski, Fulman, and the author. Our technique generalizes the Kerov's operators first appeared in [Okounkov, arXiv:math/0002135], and also stems from the study of duality of graded graphs in [Fomin, 1994]. Our main object is a countable branching graph carrying an sl(2,C)-module of a special kind. Using this structure, we introduce distinguished probability measures on the floors of the graph, and define two related types of Markov dynamics associated with these measures. We study spectral properties of the dynamics, and our main result is the explicit description of eigenfunctions of the Markov generator of one of the processes. For the Young graph our approach reconstructs the z-measures on partitions and the associated dynamics studied by Borodin and Olshanski [arXiv:math-ph/0409075, arXiv:0706.1034]. The generator of the dynamics of [arXiv:math-ph/0409075] is diagonal in the basis of the Meixner symme...

  15. Markov骨架过程成为Markov过程的条件%A Condition of a Markov Skeleton Process Being a Markov Process

    Institute of Scientific and Technical Information of China (English)

    刘万荣

    2002-01-01

    A sufficient condition of a homogeneous Markov skeleton process being a homogeneous Markov process is given and the transition function is obtained.%给出了一个齐次Markov骨架过程成为齐次Markov过程的充分条件及其转移概率函数.

  16. Testing the Adequacy of a Semi-Markov Process

    Science.gov (United States)

    2015-09-17

    metrics in Equations 2.2 and 2.3 are applications of the generalized Pearson Chi-squared test often used for assessing goodness of fit [20]. The test...1961, pp. 1243–1259. 66 [16] He, S.-w., Wang, J.-g., and Yan, J.-a., Semimartingale Theory and Stochastic Calculus , Taylor & Francis, 1992. [17...Markov processes,” Applied Statistics, 1988, pp. 242–250. [19] Aguirre-Hernández, R. and Farewell, V., “A Pearson -type goodness-of-fit test for

  17. Markov decision processes with iterated coherent risk measures

    Science.gov (United States)

    Chu, Shanyun; Zhang, Yi

    2014-11-01

    This paper considers a Markov decision process in Borel state and action spaces with the aggregated (or say iterated) coherent risk measure to be minimised. For this problem, we establish the Bellman optimality equation as well as the value and policy iteration algorithms, and show the existence of a deterministic stationary optimal policy. The cost function, while being allowed to be unbounded from below (in the sense that its negative part needs be bounded by some nonnegative real-valued possibly unbounded weight function), can be arbitrarily unbounded from above and possibly infinitely valued.

  18. Markov decision processes and the belief-desire-intention model

    CERN Document Server

    Simari, Gerardo I

    2011-01-01

    In this work, we provide a treatment of the relationship between two models that have been widely used in the implementation of autonomous agents: the Belief DesireIntention (BDI) model and Markov Decision Processes (MDPs). We start with an informal description of the relationship, identifying the common features of the two approaches and the differences between them. Then we hone our understanding of these differences through an empirical analysis of the performance of both models on the TileWorld testbed. This allows us to show that even though the MDP model displays consistently better beha

  19. Fluctuations in Markov Processes Time Symmetry and Martingale Approximation

    CERN Document Server

    Komorowski, Tomasz; Olla, Stefano

    2012-01-01

    The present volume contains the most advanced theories on the martingale approach to central limit theorems. Using the time symmetry properties of the Markov processes, the book develops the techniques that allow us to deal with infinite dimensional models that appear in statistical mechanics and engineering (interacting particle systems, homogenization in random environments, and diffusion in turbulent flows, to mention just a few applications). The first part contains a detailed exposition of the method, and can be used as a text for graduate courses. The second concerns application to exclu

  20. Synthesis for PCTL in Parametric Markov Decision Processes

    DEFF Research Database (Denmark)

    Hahn, Ernst Moritz; Han, Tingting; Zhang, Lijun

    2011-01-01

    In parametric Markov decision processes (PMDPs), transition probabilities are not fixed, but are given as functions over a set of parameters. A PMDP denotes a family of concrete MDPs. This paper studies the synthesis problem for PCTL in PMDPs: Given a specification Φ in PCTL, we synthesise...... by hyper-rectangles, we allow a limited area to remain undecided. We also consider an extension of PCTL with reachability rewards. To demonstrate the applicability of the approach, we apply our technique on a case study, using a preliminary implementation....

  1. A Learning Based Approach to Control Synthesis of Markov Decision Processes for Linear Temporal Logic Specifications

    Science.gov (United States)

    2014-09-20

    A Learning Based Approach to Control Synthesis of Markov Decision Processes for Linear Temporal Logic Specifications Dorsa Sadigh Eric Kim Samuel...2014 to 00-00-2014 4. TITLE AND SUBTITLE A Learning Based Approach to Control Synthesis of Markov Decision Processes for Linear Temporal Logic...ABSTRACT We propose to synthesize a control policy for a Markov decision process (MDP) such that the resulting traces of the MDP satisfy a linear

  2. Markov modulated Poisson process models incorporating covariates for rainfall intensity.

    Science.gov (United States)

    Thayakaran, R; Ramesh, N I

    2013-01-01

    Time series of rainfall bucket tip times at the Beaufort Park station, Bracknell, in the UK are modelled by a class of Markov modulated Poisson processes (MMPP) which may be thought of as a generalization of the Poisson process. Our main focus in this paper is to investigate the effects of including covariate information into the MMPP model framework on statistical properties. In particular, we look at three types of time-varying covariates namely temperature, sea level pressure, and relative humidity that are thought to be affecting the rainfall arrival process. Maximum likelihood estimation is used to obtain the parameter estimates, and likelihood ratio tests are employed in model comparison. Simulated data from the fitted model are used to make statistical inferences about the accumulated rainfall in the discrete time interval. Variability of the daily Poisson arrival rates is studied.

  3. CRITERIA OF STRONG TRANSIENCE FOR OPERATOR-SELF-SIMILAR MARKOV PROCESSES

    Institute of Scientific and Technical Information of China (English)

    2006-01-01

    Yamamuro in [1] defines strong and weak transience of Markov processes; gives a criterion for strong transience of Feller processes; and further, discusses strong and weak transience of Ornstein-Uhlenbeck type processes. In this article, the authors weaken the Feller property of the result in [1] to weak Feller property and discuss the strong transience of operator-self-similar Markov processes.

  4. Strategy optimization for controlled Markov process with descriptive complexity constraint

    Institute of Scientific and Technical Information of China (English)

    JIA QingShan; ZHAO QianChuan

    2009-01-01

    Due to various advantages in storage and Implementation,simple strategies are usually preferred than complex strategies when the performances are close.Strategy optimization for controlled Markov process with descriptive complexity constraint provides a general framework for many such problems.In this paper,we first show by examples that the descriptive complexity and the performance of a strategy could be Independent,and use the F-matrix in the No-Free-Lunch Theorem to show the risk that approximating complex strategies may lead to simple strategies that are unboundedly worse in cardinal performance than the original complex strategies.We then develop a method that handles the descriptive complexity constraint directly,which describes simple strategies exactly and only approximates complex strategies during the optimization.The ordinal performance difference between the resulting strategies of this selective approximation method and the global optimum is quantified.Numerical examples on an engine maintenance problem show how this method Improves the solution quality.We hope this work sheds some insights to solving general strategy optimization for controlled Markov procase with descriptive complexity constraint.

  5. The application of Markov decision process in restaurant delivery robot

    Science.gov (United States)

    Wang, Yong; Hu, Zhen; Wang, Ying

    2017-05-01

    As the restaurant delivery robot is often in a dynamic and complex environment, including the chairs inadvertently moved to the channel and customers coming and going. The traditional path planning algorithm is not very ideal. To solve this problem, this paper proposes the Markov dynamic state immediate reward (MDR) path planning algorithm according to the traditional Markov decision process. First of all, it uses MDR to plan a global path, then navigates along this path. When the sensor detects there is no obstructions in front state, increase its immediate state reward value; when the sensor detects there is an obstacle in front, plan a global path that can avoid obstacle with the current position as the new starting point and reduce its state immediate reward value. This continues until the target is reached. When the robot learns for a period of time, it can avoid those places where obstacles are often present when planning the path. By analyzing the simulation experiment, the algorithm has achieved good results in the global path planning under the dynamic environment.

  6. Upscaling of Mixing Processes using a Spatial Markov Model

    Science.gov (United States)

    Bolster, Diogo; Sund, Nicole; Porta, Giovanni

    2016-11-01

    The Spatial Markov model is a model that has been used to successfully upscale transport behavior across a broad range of spatially heterogeneous flows, with most examples to date coming from applications relating to porous media. In its most common current forms the model predicts spatially averaged concentrations. However, many processes, including for example chemical reactions, require an adequate understanding of mixing below the averaging scale, which means that knowledge of subscale fluctuations, or closures that adequately describe them, are needed. Here we present a framework, consistent with the Spatial Markov modeling framework, that enables us to do this. We apply and present it as applied to a simple example, a spatially periodic flow at low Reynolds number. We demonstrate that our upscaled model can successfully predict mixing by comparing results from direct numerical simulations to predictions with our upscaled model. To this end we focus on predicting two common metrics of mixing: the dilution index and the scalar dissipation. For both metrics our upscaled predictions very closely match observed values from the DNS. This material is based upon work supported by NSF Grants EAR-1351625 and EAR-1417264.

  7. Stochastic model of milk homogenization process using Markov's chain

    Directory of Open Access Journals (Sweden)

    A. A. Khvostov

    2016-01-01

    Full Text Available The process of development of a mathematical model of the process of homogenization of dairy products is considered in the work. The theory of Markov's chains was used in the development of the mathematical model, Markov's chain with discrete states and continuous parameter for which the homogenisation pressure is taken, being the basis for the model structure. Machine realization of the model is implemented in the medium of structural modeling MathWorks Simulink™. Identification of the model parameters was carried out by minimizing the standard deviation calculated from the experimental data for each fraction of dairy products fat phase. As the set of experimental data processing results of the micrographic images of fat globules of whole milk samples distribution which were subjected to homogenization at different pressures were used. Pattern Search method was used as optimization method with the Latin Hypercube search algorithm from Global Optimization Тoolbox library. The accuracy of calculations averaged over all fractions of 0.88% (the relative share of units, the maximum relative error was 3.7% with the homogenization pressure of 30 MPa, which may be due to the very abrupt change in properties from the original milk in the particle size distribution at the beginning of the homogenization process and the lack of experimental data at homogenization pressures of below the specified value. The mathematical model proposed allows to calculate the profile of volume and mass distribution of the fat phase (fat globules in the product, depending on the homogenization pressure and can be used in the laboratory and research of dairy products composition, as well as in the calculation, design and modeling of the process equipment of the dairy industry enterprises.

  8. Deciding when to intervene: a Markov decision process approach.

    Science.gov (United States)

    Magni, P; Quaglini, S; Marchetti, M; Barosi, G

    2000-12-01

    The aim of this paper is to point out the difference between static and dynamic approaches to choosing the optimal time for intervention. The paper demonstrates that classical approaches, such as decision trees and influence diagrams, hardly cope with dynamic problems: they cannot simulate all the real-world strategies and consequently can only calculate suboptimal solutions. A dynamic formalism based on Markov decision processes (MPPs) is then proposed and applied to a medical problem: the prophylactic surgery in mild hereditary spherocytosis. The paper compares the proposed approach with a static approach on the same medical problem. The policy provided by the dynamic approach achieved significant gain over the static policy by delaying the intervention time in some categories of patients. The calculations are carried out with DT-Planner, a graphical decision aid specifically built for dealing with dynamic decision processes.

  9. Pavement maintenance optimization model using Markov Decision Processes

    Science.gov (United States)

    Mandiartha, P.; Duffield, C. F.; Razelan, I. S. b. M.; Ismail, A. b. H.

    2017-09-01

    This paper presents an optimization model for selection of pavement maintenance intervention using a theory of Markov Decision Processes (MDP). There are some particular characteristics of the MDP developed in this paper which distinguish it from other similar studies or optimization models intended for pavement maintenance policy development. These unique characteristics include a direct inclusion of constraints into the formulation of MDP, the use of an average cost method of MDP, and the policy development process based on the dual linear programming solution. The limited information or discussions that are available on these matters in terms of stochastic based optimization model in road network management motivates this study. This paper uses a data set acquired from road authorities of state of Victoria, Australia, to test the model and recommends steps in the computation of MDP based stochastic optimization model, leading to the development of optimum pavement maintenance policy.

  10. Approximation of a class of Markov-modulated Poisson processes with a large state space

    Energy Technology Data Exchange (ETDEWEB)

    Sitaraman, H.

    1989-01-01

    Many queueing systems have an arrival process that can be modeled by a Markov-modulated Poisson process. The Markov-modulated Poisson process (MMPP) is a doubly stochastic Poisson process in which the arrival rate varies according to a finite state irreducible Markov process. In many applications of MMPPs, the point process is constructed by superpositions or similar constructions, which lead to modulating Markov processes with a large state space. Since this limits the feasibility of numerical computations, a useful problem is to approximate an MMPP represented by a large Markov process by one with fewer states. The author focuses his attention in particular, to approximating a simple but useful special case of the MMPP, namely the Birth and Death Modulated Poisson process. In the validation stage, the quality of the approximation is examined in relation to the MMPP/G/1 queue.

  11. The Hierarchical Dirichlet Process Hidden Semi-Markov Model

    CERN Document Server

    Johnson, Matthew J

    2012-01-01

    There is much interest in the Hierarchical Dirichlet Process Hidden Markov Model (HDP-HMM) as a natural Bayesian nonparametric extension of the traditional HMM. However, in many settings the HDP-HMM's strict Markovian constraints are undesirable, particularly if we wish to learn or encode non-geometric state durations. We can extend the HDP-HMM to capture such structure by drawing upon explicit-duration semi- Markovianity, which has been developed in the parametric setting to allow construction of highly interpretable models that admit natural prior information on state durations. In this paper we introduce the explicitduration HDP-HSMM and develop posterior sampling algorithms for efficient inference in both the direct-assignment and weak-limit approximation settings. We demonstrate the utility of the model and our inference methods on synthetic data as well as experiments on a speaker diarization problem and an example of learning the patterns in Morse code.

  12. On Characterisation of Markov Processes Via Martingale Problems

    Indian Academy of Sciences (India)

    Abhay G Bhatt; Rajeeva L Karandikar; B V Rao

    2006-02-01

    It is well-known that well-posedness of a martingale problem in the class of continuous (or r.c.l.l.) solutions enables one to construct the associated transition probability functions. We extend this result to the case when the martingale problem is well-posed in the class of solutions which are continuous in probability. This extension is used to improve on a criterion for a probability measure to be invariant for the semigroup associated with the Markov process. We also give examples of martingale problems that are well-posed in the class of solutions which are continuous in probability but for which no r.c.l.l. solution exists.

  13. Interacting discrete Markov processes with power-law probability distributions

    Science.gov (United States)

    Ridley, Kevin D.; Jakeman, Eric

    2017-09-01

    During recent years there has been growing interest in the occurrence of long-tailed distributions, also known as heavy-tailed or fat-tailed distributions, which can exhibit power-law behaviour and often characterise physical systems that undergo very large fluctuations. In this paper we show that the interaction between two discrete Markov processes naturally generates a time-series characterised by such a distribution. This possibility is first demonstrated by numerical simulation and then confirmed by a mathematical analysis that enables the parameter range over which the power-law occurs to be quantified. The results are supported by comparison of numerical results with theoretical predictions and general conclusions are drawn regarding mechanisms that can cause this behaviour.

  14. Regret-based Reward Elicitation for Markov Decision Processes

    CERN Document Server

    Regan, Kevin

    2012-01-01

    The specification of aMarkov decision process (MDP) can be difficult. Reward function specification is especially problematic; in practice, it is often cognitively complex and time-consuming for users to precisely specify rewards. This work casts the problem of specifying rewards as one of preference elicitation and aims to minimize the degree of precision with which a reward function must be specified while still allowing optimal or near-optimal policies to be produced. We first discuss how robust policies can be computed for MDPs given only partial reward information using the minimax regret criterion. We then demonstrate how regret can be reduced by efficiently eliciting reward information using bound queries, using regret-reduction as a means for choosing suitable queries. Empirical results demonstrate that regret-based reward elicitation offers an effective way to produce near-optimal policies without resorting to the precise specification of the entire reward function.

  15. Simulation-based algorithms for Markov decision processes

    CERN Document Server

    Chang, Hyeong Soo; Fu, Michael C; Marcus, Steven I

    2013-01-01

    Markov decision process (MDP) models are widely used for modeling sequential decision-making problems that arise in engineering, economics, computer science, and the social sciences.  Many real-world problems modeled by MDPs have huge state and/or action spaces, giving an opening to the curse of dimensionality and so making practical solution of the resulting models intractable.  In other cases, the system of interest is too complex to allow explicit specification of some of the MDP model parameters, but simulation samples are readily available (e.g., for random transitions and costs). For these settings, various sampling and population-based algorithms have been developed to overcome the difficulties of computing an optimal solution in terms of a policy and/or value function.  Specific approaches include adaptive sampling, evolutionary policy iteration, evolutionary random policy search, and model reference adaptive search. This substantially enlarged new edition reflects the latest developments in novel ...

  16. Subensemble decomposition and Markov process analysis of Burgers turbulence.

    Science.gov (United States)

    Zhang, Zhi-Xiong; She, Zhen-Su

    2011-08-01

    A numerical and statistical study is performed to describe the positive and negative local subgrid energy fluxes in the one-dimensional random-force-driven Burgers turbulence (Burgulence). We use a subensemble method to decompose the field into shock wave and rarefaction wave subensembles by group velocity difference. We observe that the shock wave subensemble shows a strong intermittency which dominates the whole Burgulence field, while the rarefaction wave subensemble satisfies the Kolmogorov 1941 (K41) scaling law. We calculate the two subensemble probabilities and find that in the inertial range they maintain scale invariance, which is the important feature of turbulence self-similarity. We reveal that the interconversion of shock and rarefaction waves during the equation's evolution displays in accordance with a Markov process, which has a stationary transition probability matrix with the elements satisfying universal functions and, when the time interval is much greater than the corresponding characteristic value, exhibits the scale-invariant property.

  17. Performance evaluation:= (process algebra + model checking) x Markov chains

    NARCIS (Netherlands)

    Hermanns, H.; Katoen, J.P.; Larsen, Kim G.; Nielsen, Mogens

    2001-01-01

    Markov chains are widely used in practice to determine system performance and reliability characteristics. The vast majority of applications considers continuous-time Markov chains (CTMCs). This tutorial paper shows how successful model specification and analysis techniques from concurrency theory c

  18. A generalized ARFIMA process with Markov-switching fractional differencing parameter

    OpenAIRE

    Tsay, Wen-Jen; Härdle, Wolfgang Karl

    2007-01-01

    We propose a general class of Markov-switching-ARFIMA processes in order to combine strands of long memory and Markov-switching literature. Although the coverage of this class of models is broad, we show that these models can be easily estimated with the DLV algorithm proposed. This algorithm combines the Durbin-Levinson and Viterbi procedures. A Monte Carlo experiment reveals that the finite sample performance of the proposed algorithm for a simple mixture model of Markov-switching mean and ...

  19. Markov vs. Hurst-Kolmogorov behaviour identification in hydroclimatic processes

    Science.gov (United States)

    Dimitriadis, Panayiotis; Gournari, Naya; Koutsoyiannis, Demetris

    2016-04-01

    Hydroclimatic processes are usually modelled either by exponential decay of the autocovariance function, i.e., Markovian behaviour, or power type decay, i.e., long-term persistence (or else Hurst-Kolmogorov behaviour). For the identification and quantification of such behaviours several graphical stochastic tools can be used such as the climacogram (i.e., plot of the variance of the averaged process vs. scale), autocovariance, variogram, power spectrum etc. with the former usually exhibiting smaller statistical uncertainty as compared to the others. However, most methodologies including these tools are based on the expected value of the process. In this analysis, we explore a methodology that combines both the practical use of a graphical representation of the internal structure of the process as well as the statistical robustness of the maximum-likelihood estimation. For validation and illustration purposes, we apply this methodology to fundamental stochastic processes, such as Markov and Hurst-Kolmogorov type ones. Acknowledgement: This research is conducted within the frame of the undergraduate course "Stochastic Methods in Water Resources" of the National Technical University of Athens (NTUA). The School of Civil Engineering of NTUA provided moral support for the participation of the students in the Assembly.

  20. The One-Dimensional Distribution and Construction of Semi-Markov Processes%半马氏过程的一维分布及构造

    Institute of Scientific and Technical Information of China (English)

    唐荣; 郭先平; 刘再明

    2008-01-01

    In this paper, we obtain the transition probability of jump chain of semi-Markov process, the distribution of sojourn time and one-dimensional distribution of semi-Markov process.Furthermore, the semi-Markov process X(t, ω) is constructed from the semi-Markov matrix and it is proved that two definitions of semi-Markov process are equivalent.

  1. POISSON TRAFFIC PROCESSES IN PURE JUMP MARKOV PROCESSES AND GENERALIZED NETWORKS

    Institute of Scientific and Technical Information of China (English)

    CAO Chengxuan; XU Guanghui

    2001-01-01

    In this paper, we present the conditions under which the traffic processes in a pure jump Markov process with a general state space are Poisson processes, and give a simple proof of PASTA type theorem in Melamed (1982) and Walrand (1988).Furthermore, we consider a generalized network with phase type negative arrivals and show that the network has a product-form invariant distribution and its traffic processes which represent the customers exiting from the network are Poisson processes.

  2. Efficient computation of time-bounded reachability probabilities in uniform continuous-time Markov decision processes

    NARCIS (Netherlands)

    Baier, Christel; Hermanns, H.; Katoen, Joost P.; Haverkort, Boudewijn R.H.M.

    2005-01-01

    A continuous-time Markov decision process (CTMDP) is a generalization of a continuous-time Markov chain in which both probabilistic and nondeterministic choices co-exist. This paper presents an efficient algorithm to compute the maximum (or minimum) probability to reach a set of goal states within a

  3. Non-equilibrium Thermodynamics of Piecewise Deterministic Markov Processes

    Science.gov (United States)

    Faggionato, A.; Gabrielli, D.; Ribezzi Crivellari, M.

    2009-10-01

    We consider a class of stochastic dynamical systems, called piecewise deterministic Markov processes, with states ( x, σ)∈Ω×Γ, Ω being a region in ℝ d or the d-dimensional torus, Γ being a finite set. The continuous variable x follows a piecewise deterministic dynamics, the discrete variable σ evolves by a stochastic jump dynamics and the two resulting evolutions are fully-coupled. We study stationarity, reversibility and time-reversal symmetries of the process. Increasing the frequency of the σ-jumps, the system behaves asymptotically as deterministic and we investigate the structure of its fluctuations (i.e. deviations from the asymptotic behavior), recovering in a non Markovian frame results obtained by Bertini et al. (Phys. Rev. Lett. 87(4):040601, 2001; J. Stat. Phys. 107(3-4):635-675, 2002; J. Stat. Mech. P07014, 2007; Preprint available online at http://www.arxiv.org/abs/0807.4457, 2008), in the context of Markovian stochastic interacting particle systems. Finally, we discuss a Gallavotti-Cohen-type symmetry relation with involution map different from time-reversal.

  4. On the Empty Essential Spectrum for Markov Processes in Dimension One

    Institute of Scientific and Technical Information of China (English)

    Yong Hua MAO

    2006-01-01

    This paper gives characterizations for diffusion processes on the line and birth-death processes whose generators admit the empty essential spectra. Some equivalent conditions for empty essential spectra for general Markov generators are also discussed.

  5. A Minimum Relative Entropy Controller for Undiscounted Markov Decision Processes

    CERN Document Server

    Ortega, Pedro A

    2010-01-01

    Adaptive control problems are notoriously difficult to solve even in the presence of plant-specific controllers. One way to by-pass the intractable computation of the optimal policy is to restate the adaptive control as the minimization of the relative entropy of a controller that ignores the true plant dynamics from an informed controller. The solution is given by the Bayesian control rule-a set of equations characterizing a stochastic adaptive controller for the class of possible plant dynamics. Here, the Bayesian control rule is applied to derive BCR-MDP, a controller to solve undiscounted Markov decision processes with finite state and action spaces and unknown dynamics. In particular, we derive a non-parametric conjugate prior distribution over the policy space that encapsulates the agent's whole relevant history and we present a Gibbs sampler to draw random policies from this distribution. Preliminary results show that BCR-MDP successfully avoids sub-optimal limit cycles due to its built-in mechanism to...

  6. Multi-Objective Model Checking of Markov Decision Processes

    CERN Document Server

    Etessami, Kousha; Vardi, Moshe Y; Yannakakis, Mihalis

    2008-01-01

    We study and provide efficient algorithms for multi-objective model checking problems for Markov Decision Processes (MDPs). Given an MDP, $M$, and given multiple linear-time ($\\omega$-regular or LTL) properties $\\varphi_i$, and probabilities $r_i \\in [0,1]$, $i=1,...,k$, we ask whether there exists a strategy $\\sigma$ for the controller such that, for all $i$, the probability that a trajectory of $M$ controlled by $\\sigma$ satisfies $\\varphi_i$ is at least $r_i$. We provide an algorithm that decides whether there exists such a strategy and if so produces it, and which runs in time polynomial in the size of the MDP. Such a strategy may require the use of both randomization and memory. We also consider more general multi-objective $\\omega$-regular queries, which we motivate with an application to assume-guarantee compositional reasoning for probabilistic systems. Note that there can be trade-offs between different properties: satisfying property $\\varphi_1$ with high probability may necessitate satisfying $\\var...

  7. Analysing grouping of nucleotides in DNA sequences using lumped processes constructed from Markov chains.

    Science.gov (United States)

    Guédon, Yann; d'Aubenton-Carafa, Yves; Thermes, Claude

    2006-03-01

    The most commonly used models for analysing local dependencies in DNA sequences are (high-order) Markov chains. Incorporating knowledge relative to the possible grouping of the nucleotides enables to define dedicated sub-classes of Markov chains. The problem of formulating lumpability hypotheses for a Markov chain is therefore addressed. In the classical approach to lumpability, this problem can be formulated as the determination of an appropriate state space (smaller than the original state space) such that the lumped chain defined on this state space retains the Markov property. We propose a different perspective on lumpability where the state space is fixed and the partitioning of this state space is represented by a one-to-many probabilistic function within a two-level stochastic process. Three nested classes of lumped processes can be defined in this way as sub-classes of first-order Markov chains. These lumped processes enable parsimonious reparameterizations of Markov chains that help to reveal relevant partitions of the state space. Characterizations of the lumped processes on the original transition probability matrix are derived. Different model selection methods relying either on hypothesis testing or on penalized log-likelihood criteria are presented as well as extensions to lumped processes constructed from high-order Markov chains. The relevance of the proposed approach to lumpability is illustrated by the analysis of DNA sequences. In particular, the use of lumped processes enables to highlight differences between intronic sequences and gene untranslated region sequences.

  8. Coupling, convergence rates of Markov processes and weak Poincaré inequalities

    Institute of Scientific and Technical Information of China (English)

    WANG; Fengyu(王凤雨)

    2002-01-01

    Some analytic and probabilistic properties of the weak Poincaré inequality are obtained. In particular, for strong Feller Markov processes the existence of this inequality is equivalent to each of the following: (i)the Liouville property (or the irreducibility); (ii) the existence of successful couplings (or shift-couplings); (iii)the convergence of the Markov process in total variation norm; (iv) the triviality of the tail (or the invariant)σ-field; (v) the convergence of the density. Estimates of the convergence rate in total variation norm of Markov processes are obtained using the weak Poincaré inequality.

  9. Nonlinear Stochastic H∞ Control with Markov Jumps and (x,u,v-Dependent Noise: Finite and Infinite Horizon Cases

    Directory of Open Access Journals (Sweden)

    Li Sheng

    2014-01-01

    Full Text Available This paper is concerned with the H∞ control problem for nonlinear stochastic Markov jump systems with state, control, and external disturbance-dependent noise. By means of inequality techniques and coupled Hamilton-Jacobi inequalities, both finite and infinite horizon H∞ control designs of such systems are developed. Two numerical examples are provided to illustrate the effectiveness of the proposed design method.

  10. Neural Network Based Finite-Time Stabilization for Discrete-Time Markov Jump Nonlinear Systems with Time Delays

    Directory of Open Access Journals (Sweden)

    Fei Chen

    2013-01-01

    Full Text Available This paper deals with the finite-time stabilization problem for discrete-time Markov jump nonlinear systems with time delays and norm-bounded exogenous disturbance. The nonlinearities in different jump modes are parameterized by neural networks. Subsequently, a linear difference inclusion state space representation for a class of neural networks is established. Based on this, sufficient conditions are derived in terms of linear matrix inequalities to guarantee stochastic finite-time boundedness and stochastic finite-time stabilization of the closed-loop system. A numerical example is illustrated to verify the efficiency of the proposed technique.

  11. On the optimality equation for average cost Markov control processes with Feller transition probabilities

    Science.gov (United States)

    Jaskiewicz, Anna; Nowak, Andrzej S.

    2006-04-01

    We consider Markov control processes with Borel state space and Feller transition probabilities, satisfying some generalized geometric ergodicity conditions. We provide a new theorem on the existence of a solution to the average cost optimality equation.

  12. Students' Progress throughout Examination Process as a Markov Chain

    Science.gov (United States)

    Hlavatý, Robert; Dömeová, Ludmila

    2014-01-01

    The paper is focused on students of Mathematical methods in economics at the Czech university of life sciences (CULS) in Prague. The idea is to create a model of students' progress throughout the whole course using the Markov chain approach. Each student has to go through various stages of the course requirements where his success depends on the…

  13. First passage process of a Markov additive process, with applications to reflection problems

    CERN Document Server

    D'Auria, Bernardo; Kella, Offer; Mandjes, Michel

    2010-01-01

    In this paper we consider the first passage process of a spectrally negative Markov additive process (MAP). The law of this process is uniquely characterized by a certain matrix function, which plays a crucial role in fluctuation theory. We show how to identify this matrix using the theory of Jordan chains associated with analytic matrix functions. Importantly, our result also provides us with a technique, which can be used to derive various further identities. We then proceed to show how to compute the stationary distribution associated with a one-sided reflected (at zero) MAP for both the spectrally positive and spectrally negative cases as well as for the two sided reflected Markov-modulated Brownian motion; these results can be interpreted in terms of queues with MAP input.

  14. CMOS Nonlinear Signal Processing Circuits

    OpenAIRE

    2010-01-01

    The chapter describes various nonlinear signal processing CMOS circuits, including a high reliable WTA/LTA, simple MED cell, and low-voltage arbitrary order extractor. We focus the discussion on CMOS analog circuit design with reliable, programmable capability, and low voltage operation. It is a practical problem when the multiple identical cells are required to match and realized within a single chip using a conventional process. Thus, the design of high-reliable circuit is indeed needed. Th...

  15. Compensating Operator and Weak Convergence of Semi-Markov Process to the Diffusion Process without Balance Condition

    Directory of Open Access Journals (Sweden)

    Igor V. Malyk

    2015-01-01

    Full Text Available Weak convergence of semi-Markov processes in the diffusive approximation scheme is studied in the paper. This problem is not new and it is studied in many papers, using convergence of random processes. Unlike other studies, we used in this paper concept of the compensating operator. It enables getting sufficient conditions of weak convergence under the conditions on the local characteristics of output semi-Markov process.

  16. Dynamic response of mechanical systems to impulse process stochastic excitations: Markov approach

    Science.gov (United States)

    Iwankiewicz, R.

    2016-05-01

    Methods for determination of the response of mechanical dynamic systems to Poisson and non-Poisson impulse process stochastic excitations are presented. Stochastic differential and integro-differential equations of motion are introduced. For systems driven by Poisson impulse process the tools of the theory of non-diffusive Markov processes are used. These are: the generalized Itô’s differential rule which allows to derive the differential equations for response moments and the forward integro-differential Chapman-Kolmogorov equation from which the equation governing the probability density of the response is obtained. The relation of Poisson impulse process problems to the theory of diffusive Markov processes is given. For systems driven by a class of non-Poisson (Erlang renewal) impulse processes an exact conversion of the original non-Markov problem into a Markov one is based on the appended Markov chain corresponding to the introduced auxiliary pure jump stochastic process. The derivation of the set of integro-differential equations for response probability density and also a moment equations technique are based on the forward integro-differential Chapman-Kolmogorov equation. An illustrating numerical example is also included.

  17. On Determining the Order of Markov Dependence of an Observed Process Governed by a Hidden Markov Model

    Directory of Open Access Journals (Sweden)

    R.J. Boys

    2002-01-01

    Full Text Available This paper describes a Bayesian approach to determining the order of a finite state Markov chain whose transition probabilities are themselves governed by a homogeneous finite state Markov chain. It extends previous work on homogeneous Markov chains to more general and applicable hidden Markov models. The method we describe uses a Markov chain Monte Carlo algorithm to obtain samples from the (posterior distribution for both the order of Markov dependence in the observed sequence and the other governing model parameters. These samples allow coherent inferences to be made straightforwardly in contrast to those which use information criteria. The methods are illustrated by their application to both simulated and real data sets.

  18. 模型不确定非线性Markov跳变系统的滤波算法%Filter algorithm for nonlinear Markov jump systems with uncertain models

    Institute of Scientific and Technical Information of China (English)

    赵顺毅; 刘飞

    2012-01-01

    Considering the state estimation problem for the nonlinear Markov jump system with uncertain model, a novel filtering algorithm is proposed. Compared with the traditional interacting multiple particle filter method, in this method, a term of filtering error at previous time instant is introduced to increase the effect of the particles which are true but with small weights due to the inaccuracy model to improve the estimation performance in the filtering process. Simulation results show the effectiveness of this method in handling with the state estimation problem for the nonlinear Markov jump systems with uncertain model parameter.%针对模型不确定非线性Markov跳变系统,提出一种新的滤波算法.相比于传统交互多模型粒子滤波,该方法通过引入前一时刻的滤波误差来增强原先由于不精确模型而造成权值较小的真实粒子在滤波过程中的作用,以此来改善算法的估计性能.仿真结果表明,该方法在处理含不确定模型参数的非线性Markov跳变系统状态估计问题时具有较好的性能.

  19. Towards a Theory of Sampled-Data Piecewise-Deterministic Markov Processes

    Science.gov (United States)

    Herencia-Zapana, Heber; Gonzalez, Oscar R.; Gray, W. Steven

    2006-01-01

    The analysis and design of practical control systems requires that stochastic models be employed. Analysis and design tools have been developed, for example, for Markovian jump linear continuous and discrete-time systems, piecewise-deterministic processes (PDP's), and general stochastic hybrid systems (GSHS's). These model classes have been used in many applications, including fault tolerant control and networked control systems. This paper presents initial results on the analysis of a sampled-data PDP representation of a nonlinear sampled-data system with a jump linear controller. In particular, it is shown that the state of the sampled-data PDP satisfies the strong Markov property. In addition, a relation between the invariant measures of a sampled-data system driven by a stochastic process and its associated discrete-time representation are presented. As an application, when the plant is linear with no external input, a sufficient testable condition for the convergence in distribution to the invariant delta Dirac measure is given.

  20. Sharp Bounds for the First Eigenvalue of Symmetric Markov Processes and Their Applications

    Institute of Scientific and Technical Information of China (English)

    Jian WANG

    2012-01-01

    By adopting a nice auxiliary transform of Markov operators,we derive new bounds for the first eigenvalue of the generator corresponding to symmetric Markov processes.Our results not only extend the related topic in the literature,but also are efficiently used to study the first eigenvalue of birth-death processes with killing and that of elliptic operators with killing on half line.In particular,we obtain two approximation procedures for the first eigenvalue of birth-death processes with killing,and present qualitatively sharp upper and lower bounds for the first eigenvalue of elliptic operators with killing on half line.

  1. Markov decision processes: a tool for sequential decision making under uncertainty.

    Science.gov (United States)

    Alagoz, Oguzhan; Hsu, Heather; Schaefer, Andrew J; Roberts, Mark S

    2010-01-01

    We provide a tutorial on the construction and evaluation of Markov decision processes (MDPs), which are powerful analytical tools used for sequential decision making under uncertainty that have been widely used in many industrial and manufacturing applications but are underutilized in medical decision making (MDM). We demonstrate the use of an MDP to solve a sequential clinical treatment problem under uncertainty. Markov decision processes generalize standard Markov models in that a decision process is embedded in the model and multiple decisions are made over time. Furthermore, they have significant advantages over standard decision analysis. We compare MDPs to standard Markov-based simulation models by solving the problem of the optimal timing of living-donor liver transplantation using both methods. Both models result in the same optimal transplantation policy and the same total life expectancies for the same patient and living donor. The computation time for solving the MDP model is significantly smaller than that for solving the Markov model. We briefly describe the growing literature of MDPs applied to medical decisions.

  2. The application of Markov decision process with penalty function in restaurant delivery robot

    Science.gov (United States)

    Wang, Yong; Hu, Zhen; Wang, Ying

    2017-05-01

    As the restaurant delivery robot is often in a dynamic and complex environment, including the chairs inadvertently moved to the channel and customers coming and going. The traditional Markov decision process path planning algorithm is not save, the robot is very close to the table and chairs. To solve this problem, this paper proposes the Markov Decision Process with a penalty term called MDPPT path planning algorithm according to the traditional Markov decision process (MDP). For MDP, if the restaurant delivery robot bumps into an obstacle, the reward it receives is part of the current status reward. For the MDPPT, the reward it receives not only the part of the current status but also a negative constant term. Simulation results show that the MDPPT algorithm can plan a more secure path.

  3. Dual Weighted Markov Branching Processes%对偶加权Markov分支过程

    Institute of Scientific and Technical Information of China (English)

    蔡雨; 李扬荣

    2008-01-01

    研究对偶加权Markov分支过程的正则性、唯一性、单调性和Feller性, 得到了判断这些性质的充要以及充分或必要条件.%This paper focuses on discussing some basic properties of the dual weighted Markov branching processes which are by definition of a Siegmund's pre-dual of some weighted Markov branching processes. The regularity and uniqueness criteria, which are very easy to verify, are established. And the Feller property and monotonicity are obtained.

  4. Critical Age-Dependent Branching Markov Processes and their Scaling Limits

    Indian Academy of Sciences (India)

    Krishna B Athreya; Siva R Athreya; Srikanth K Iyer

    2010-06-01

    This paper studies: (i) the long-time behaviour of the empirical distribution of age and normalized position of an age-dependent critical branching Markov process conditioned on non-extinction; and (ii) the super-process limit of a sequence of age-dependent critical branching Brownian motions.

  5. Exponential and Strong Ergodicity for Markov Processes with an Application to Queues

    Institute of Scientific and Technical Information of China (English)

    Yuanyuan LIU; Zhenting HOU

    2008-01-01

    For an ergodic continuous-time Markov process with a particular state in its space, the authors provide the necessary and sufficient conditions for exponential and strongerg odicity in terms of the moments of the first hitting time on the state. An application to the queue length process of M/G/1 queue with multiple vacations is given.

  6. Unbounded-rate Markov decision processes : structural properties via a parametrisation approach

    NARCIS (Netherlands)

    Blok, H.

    2016-01-01

    This research is interested in optimal control of Markov decision processes (MDPs). Herein a key role is played by structural properties. Properties such as monotonicity and convexity help in finding the optimal policy. Value iteration is a tool to derive such properties in discrete time processes.

  7. h(h)-transforms of Positivity Preserving Semigroups and Associated Markov Processes

    Institute of Scientific and Technical Information of China (English)

    Xin Fang HAN; Zhi-Ming MA; Wei SUN

    2011-01-01

    The h(h)-transforms of positivity preserving semigroups and their associated Markov processes are investigated in this paper. In particular, it is shown that any quasi-regular positivity preserving coercive form is h(h)-associated with a pair of special standard processes which are in weak duality.

  8. Behavior of an Almost Semicontinuous Poisson Process on a Markov Chain Upon Attainment of A Level

    CERN Document Server

    Karnaukh, Ievgen

    2011-01-01

    We consider the almost semi-continuous processes defined on a finite Markov chain. The representation of the moment generating functions for the absolute maximum after achievement positive level and for the recovery time are obtained. Modified processes with two-step rate of negative jumps are investigated.

  9. Depth map calculation for a variable number of moving objects using Markov sequential object processes

    NARCIS (Netherlands)

    Lieshout, M.N.M. van

    2008-01-01

    We advocate the use of Markov sequential object processes for tracking a variable number of moving objects through video frames with a view towards depth calculation. A regression model based on a sequential object process quantifies goodness of fit; regularization terms are incorporated to control

  10. Potts model based on a Markov process computation solves the community structure problem effectively.

    Science.gov (United States)

    Li, Hui-Jia; Wang, Yong; Wu, Ling-Yun; Zhang, Junhua; Zhang, Xiang-Sun

    2012-07-01

    The Potts model is a powerful tool to uncover community structure in complex networks. Here, we propose a framework to reveal the optimal number of communities and stability of network structure by quantitatively analyzing the dynamics of the Potts model. Specifically we model the community structure detection Potts procedure by a Markov process, which has a clear mathematical explanation. Then we show that the local uniform behavior of spin values across multiple timescales in the representation of the Markov variables could naturally reveal the network's hierarchical community structure. In addition, critical topological information regarding multivariate spin configuration could also be inferred from the spectral signatures of the Markov process. Finally an algorithm is developed to determine fuzzy communities based on the optimal number of communities and the stability across multiple timescales. The effectiveness and efficiency of our algorithm are theoretically analyzed as well as experimentally validated.

  11. Potts model based on a Markov process computation solves the community structure problem effectively

    CERN Document Server

    Li, Hui-Jia; Wu, Ling-Yun; Zhang, Junhua; Zhang, Xiang-Sun

    2015-01-01

    Potts model is a powerful tool to uncover community structure in complex networks. Here, we propose a new framework to reveal the optimal number of communities and stability of network structure by quantitatively analyzing the dynamics of Potts model. Specifically we model the community structure detection Potts procedure by a Markov process, which has a clear mathematical explanation. Then we show that the local uniform behavior of spin values across multiple timescales in the representation of the Markov variables could naturally reveal the network's hierarchical community structure. In addition, critical topological information regarding to multivariate spin configuration could also be inferred from the spectral signatures of the Markov process. Finally an algorithm is developed to determine fuzzy communities based on the optimal number of communities and the stability across multiple timescales. The effectiveness and efficiency of our algorithm are theoretically analyzed as well as experimentally validate...

  12. Branching Markov processes on fragmentation trees generated from the paintbox process

    CERN Document Server

    Crane, Harry

    2011-01-01

    A fragmentation of a set $A$ is a graph with vertices labeled by subsets of $A$ which obey a certain parent-child relationship. A random fragmentation tree is a probability distribution on the space of fragmentations of a set. It is often convenient to regard a fragmentation tree as a collection of subsets such that each subset is associated with a non-trivial partition of itself, called its children. In this paper, we study a Markov process on the space of fragmentation trees whose transition probabilities are a product of consistent transition probabilities on the space of partitions. The result is a consistent family of transition probabilities on fragmentation trees which characterizes an infinitely exchangeable process on trees labeled by subsets of the natural numbers. We show that this process possesses a unique stationary measure and can be extended to a process on weighted trees, or trees with edge lengths, as well as mass fragmentations.

  13. Likelihood free inference for Markov processes: a comparison.

    Science.gov (United States)

    Owen, Jamie; Wilkinson, Darren J; Gillespie, Colin S

    2015-04-01

    Approaches to Bayesian inference for problems with intractable likelihoods have become increasingly important in recent years. Approximate Bayesian computation (ABC) and "likelihood free" Markov chain Monte Carlo techniques are popular methods for tackling inference in these scenarios but such techniques are computationally expensive. In this paper we compare the two approaches to inference, with a particular focus on parameter inference for stochastic kinetic models, widely used in systems biology. Discrete time transition kernels for models of this type are intractable for all but the most trivial systems yet forward simulation is usually straightforward. We discuss the relative merits and drawbacks of each approach whilst considering the computational cost implications and efficiency of these techniques. In order to explore the properties of each approach we examine a range of observation regimes using two example models. We use a Lotka-Volterra predator-prey model to explore the impact of full or partial species observations using various time course observations under the assumption of known and unknown measurement error. Further investigation into the impact of observation error is then made using a Schlögl system, a test case which exhibits bi-modal state stability in some regions of parameter space.

  14. Risk aversion and risk seeking in multicriteria forest management: a Markov decision process approach

    Science.gov (United States)

    Joseph Buongiorno; Mo Zhou; Craig Johnston

    2017-01-01

    Markov decision process models were extended to reflect some consequences of the risk attitude of forestry decision makers. One approach consisted of maximizing the expected value of a criterion subject to an upper bound on the variance or, symmetrically, minimizing the variance subject to a lower bound on the expected value.  The other method used the certainty...

  15. Strategy Complexity of Finite-Horizon Markov Decision Processes and Simple Stochastic Games

    DEFF Research Database (Denmark)

    Ibsen-Jensen, Rasmus; Chatterjee, Krishnendu

    2012-01-01

    Markov decision processes (MDPs) and simple stochastic games (SSGs) provide a rich mathematical framework to study many important problems related to probabilistic systems. MDPs and SSGs with finite-horizon objectives, where the goal is to maximize the probability to reach a target state in a given...

  16. Data-based inference of generators for Markov jump processes using convex optimization

    NARCIS (Netherlands)

    Crommelin, D.T.; Vanden-Eijnden, E.

    2009-01-01

    A variational approach to the estimation of generators for Markov jump processes from discretely sampled data is discussed and generalized. In this approach, one first calculates the spectrum of the discrete maximum likelihood estimator for the transition matrix consistent with the discrete data. Th

  17. The invariant measure of homogeneous Markov processes in the quarter-plane: Representation in geometric terms

    NARCIS (Netherlands)

    Chen, Yanting; Boucherie, Richard J.; Goseling, Jasper

    2011-01-01

    We consider the invariant measure of a homogeneous continuous-time Markov process in the quarter-plane. The basic solutions of the global balance equation are the geometric distributions. We first show that the invariant measure can not be a finite linear combination of basic geometric distributions

  18. POISSON REPRESENTATIONS OF BRANCHING MARKOV AND MEASURE-VALUED BRANCHING PROCESSES

    NARCIS (Netherlands)

    Kurtz, Thomas G.; Rodrigues, Eliane R.

    2011-01-01

    Representations of branching Markov processes and their measure-valued limits in terms of countable systems of particles are constructed for models with spatially varying birth and death rates. Each particle has a location and a "level," but unlike earlier constructions, the levels change with time.

  19. A New Vector Markov Process for M/G/1 Queue

    Institute of Scientific and Technical Information of China (English)

    严庆强; 史定华; 郭兴国

    2005-01-01

    In this paper, by considering the stochastic process of the busy period and the idle period, and introducing the unfinished work as a supplementary variable, a new vector Markov process was presented to study the M/G/1 queue again. Through establishing and solving the density evolution equations, the busy-period distribution, and the stationary distributionof waiting time and queue length were obtained. In addition, the stability condition of this queue system was given by means of an imbedded renewal process.

  20. Fault detection and diagnosis in a food pasteurization process with Hidden Markov Models

    OpenAIRE

    Tokatlı, Figen; Cinar, Ali

    2004-01-01

    Hidden Markov Models (HMM) are used to detect abnormal operation of dynamic processes and diagnose sensor and actuator faults. The method is illustrated by monitoring the operation of a pasteurization plant and diagnosing causes of abnormal operation. Process data collected under the influence of faults of different magnitude and duration in sensors and actuators are used to illustrate the use of HMM in the detection and diagnosis of process faults. Case studies with experimental data from a ...

  1. Nonlinear filtering for LIDAR signal processing

    Directory of Open Access Journals (Sweden)

    D. G. Lainiotis

    1996-01-01

    Full Text Available LIDAR (Laser Integrated Radar is an engineering problem of great practical importance in environmental monitoring sciences. Signal processing for LIDAR applications involves highly nonlinear models and consequently nonlinear filtering. Optimal nonlinear filters, however, are practically unrealizable. In this paper, the Lainiotis's multi-model partitioning methodology and the related approximate but effective nonlinear filtering algorithms are reviewed and applied to LIDAR signal processing. Extensive simulation and performance evaluation of the multi-model partitioning approach and its application to LIDAR signal processing shows that the nonlinear partitioning methods are very effective and significantly superior to the nonlinear extended Kalman filter (EKF, which has been the standard nonlinear filter in past engineering applications.

  2. Limit theorems for Markov processes indexed by continuous time Galton-Watson trees

    CERN Document Server

    Bansaye, Vincent; Marsalle, Laurence; Tran, Viet Chi

    2009-01-01

    We study the evolution of a particle system whose genealogy is given by a supercritical continuous time Galton-Watson tree. The particles move independently according to a Markov process and when a branching event occurs, the offspring locations depend on the position of the mother and the number of offspring. We prove a law of large numbers for the empirical measure of individuals alive at time $t$. This relies on a probabilistic interpretation of its intensity by mean of an auxiliary process. This latter has the same generator as the Markov process along the branches plus additional branching events, associated with jumps of accelerated rate and biased distribution. This comes from the fact that choosing an individual uniformly at time $t$ favors lineages with more branching events and larger offspring number. The central limit theorem is considered on a special case. Several examples are developed, including applications to splitting diffusions, cellular aging, branching L\\'evy processes and ancestral line...

  3. Open Markov processes: A compositional perspective on non-equilibrium steady states in biology

    CERN Document Server

    Pollard, Blake S

    2016-01-01

    In recent work, Baez, Fong and the author introduced a framework for describing Markov processes equipped with a detailed balanced equilibrium as open systems of a certain type. These `open Markov processes' serve as the building blocks for more complicated processes. In this paper, we describe the potential application of this framework in the modeling of biological systems as open systems maintained away from equilibrium. We show that non-equilibrium steady states emerge in open systems of this type, even when the rates of the underlying process are such that a detailed balanced equilibrium is permitted. It is shown that these non-equilibrium steady states minimize a quadratic form which we call `dissipation.' In some circumstances, the dissipation is approximately equal to the rate of change of relative entropy plus a correction term. On the other hand, Prigogine's principle of minimum entropy production generally fails for non-equilibrium steady states. We use a simple model of membrane transport to illus...

  4. A necessary and sufficient condition for gelation of a reversible Markov process of polymerization

    CERN Document Server

    Han, D

    2003-01-01

    A reversible Markov process as a chemical polymerization model which permits the coagulation and fragmentation reactions is considered. We present a necessary and sufficient condition for the occurrence of a gelation in the process. We show that a gelation transition may or may not occur, depending on the value of the fragmentation strength, and, in the case that gelation takes place, a critical value for the occurrence of the gelation and the mass of the gel can be determined by close forms.

  5. Theory of eigenvalues for periodic non-stationary Markov processes: the Kolmogorov operator and its applications

    Energy Technology Data Exchange (ETDEWEB)

    Caceres, Manuel O [Abdus Salam International Centre for Theoretical Physics, Strada Costiera, 11-34014 Trieste (Italy); Lobos, Alejandro M [Centro Atomico Bariloche, Instituto Balseiro, CNEA, Universidad Nacional de Cuyo, and CONICET, Av E Bustillo Km 9.5, 8400 Bariloche (Argentina)

    2006-02-17

    We present an eigenvalue theory to study the stochastic dynamics of non-stationary time-periodic Markov processes. The analysis is carried out by solving an integral operator of the Fredholm type, i.e. considering complex-valued functions fulfilling the Kolmogorov compatibility condition. We show that the asymptotic behaviour of the stochastic process is characterized by the smaller time-scale associated with the spectrum of the Kolmogorov operator. The presence of time-periodic elements in the evolution equation for the semigroup leads to a Floquet analysis. The first non-trivial Kolmogorov's eigenvalue is interpreted from a physical point of view. This non-trivial characteristic time-scale strongly depends on the interplay between the stochastic behaviour of the process and the time-periodic structure of the Fokker-Planck equation for continuous processes, or the periodically modulated master equation for discrete Markov processes. We present pedagogical examples in a finite-dimensional vector space to calculate the Kolmogorov characteristic time-scale for discrete Markov processes.

  6. Correlated Node Behavior Model based on Semi Markov Process for MANETS

    CERN Document Server

    Azni, A H; Noh, Zul Azri Muhamad; Basari, Abd Samad Hasan; Hussin, Burairah

    2012-01-01

    This paper introduces a new model for node behavior namely Correlated Node Behavior Model which is an extension of Node Behavior Model. The model adopts semi Markov process in continuous time which clusters the node that has correlation. The key parameter of the process is determined by five probabilistic parameters based on the Markovian model. Computed from the transition probabilities of the semi-Markov process, the node correlation impact on network survivability and resilience can be measure quantitatively. From the result, the quantitative analysis of correlated node behavior on the survivability is obtained through mathematical description, and the effectiveness and rationality of the proposed model are verified through numerical analysis. The analytical results show that the effect from correlated failure nodes on network survivability is much severer than other misbehaviors.

  7. Multiorder nonlinear diffraction in frequency doubling processes

    DEFF Research Database (Denmark)

    Saltiel, Solomon M.; Neshev, Dragomir N.; Krolikowski, Wieslaw

    2009-01-01

    We analyze experimentally light scattering from 2 nonlinear gratings and observe two types of second-harmonic frequency-scattering processes. The first process is identified as Raman–Nath type nonlinear diffraction that is explained by applying only transverse phase-matching conditions. The angular...... position of this type of diffraction is defined by the ratio of the second-harmonic wavelength and the grating period. In contrast, the second type of nonlinear scattering process is explained by the longitudinal phase matching only, being insensitive to the nonlinear grating...

  8. Generalization bounds of ERM-based learning processes for continuous-time Markov chains.

    Science.gov (United States)

    Zhang, Chao; Tao, Dacheng

    2012-12-01

    Many existing results on statistical learning theory are based on the assumption that samples are independently and identically distributed (i.i.d.). However, the assumption of i.i.d. samples is not suitable for practical application to problems in which samples are time dependent. In this paper, we are mainly concerned with the empirical risk minimization (ERM) based learning process for time-dependent samples drawn from a continuous-time Markov chain. This learning process covers many kinds of practical applications, e.g., the prediction for a time series and the estimation of channel state information. Thus, it is significant to study its theoretical properties including the generalization bound, the asymptotic convergence, and the rate of convergence. It is noteworthy that, since samples are time dependent in this learning process, the concerns of this paper cannot (at least straightforwardly) be addressed by existing methods developed under the sample i.i.d. assumption. We first develop a deviation inequality for a sequence of time-dependent samples drawn from a continuous-time Markov chain and present a symmetrization inequality for such a sequence. By using the resultant deviation inequality and symmetrization inequality, we then obtain the generalization bounds of the ERM-based learning process for time-dependent samples drawn from a continuous-time Markov chain. Finally, based on the resultant generalization bounds, we analyze the asymptotic convergence and the rate of convergence of the learning process.

  9. A fast exact simulation method for a class of Markov jump processes

    Energy Technology Data Exchange (ETDEWEB)

    Li, Yao, E-mail: yaoli@math.umass.edu [Department of Mathematics and Statistics, University of Massachusetts Amherst, Amherst, Massachusetts 10003 (United States); Hu, Lili, E-mail: lilyhu86@gmail.com [School of Mathematics, Georgia Institute of Technology, Atlanta, Georgia 30332 (United States)

    2015-11-14

    A new method of the stochastic simulation algorithm (SSA), named the Hashing-Leaping method (HLM), for exact simulations of a class of Markov jump processes, is presented in this paper. The HLM has a conditional constant computational cost per event, which is independent of the number of exponential clocks in the Markov process. The main idea of the HLM is to repeatedly implement a hash-table-like bucket sort algorithm for all times of occurrence covered by a time step with length τ. This paper serves as an introduction to this new SSA method. We introduce the method, demonstrate its implementation, analyze its properties, and compare its performance with three other commonly used SSA methods in four examples. Our performance tests and CPU operation statistics show certain advantages of the HLM for large scale problems.

  10. A fast exact simulation method for a class of Markov jump processes

    Science.gov (United States)

    Li, Yao; Hu, Lili

    2015-11-01

    A new method of the stochastic simulation algorithm (SSA), named the Hashing-Leaping method (HLM), for exact simulations of a class of Markov jump processes, is presented in this paper. The HLM has a conditional constant computational cost per event, which is independent of the number of exponential clocks in the Markov process. The main idea of the HLM is to repeatedly implement a hash-table-like bucket sort algorithm for all times of occurrence covered by a time step with length τ. This paper serves as an introduction to this new SSA method. We introduce the method, demonstrate its implementation, analyze its properties, and compare its performance with three other commonly used SSA methods in four examples. Our performance tests and CPU operation statistics show certain advantages of the HLM for large scale problems.

  11. A cyclic time-dependent Markov process to model daily patterns in wind turbine power production

    CERN Document Server

    Scholz, Teresa; Estanqueiro, Ana

    2013-01-01

    Wind energy is becoming a top contributor to the renewable energy mix, which raises potential reliability issues for the grid due to the fluctuating nature of its source. To achieve adequate reserve commitment and to promote market participation, it is necessary to provide models that can capture daily patterns in wind power production. This paper presents a cyclic inhomogeneous Markov process, which is based on a three-dimensional state-space (wind power, speed and direction). Each time-dependent transition probability is expressed as a Bernstein polynomial. The model parameters are estimated by solving a constrained optimization problem: The objective function combines two maximum likelihood estimators, one to ensure that the Markov process long-term behavior reproduces the data accurately and another to capture daily fluctuations. A convex formulation for the overall optimization problem is presented and its applicability demonstrated through the analysis of a case-study. The proposed model is capable of r...

  12. Nonuniqueness versus Uniqueness of Optimal Policies in Convex Discounted Markov Decision Processes

    Directory of Open Access Journals (Sweden)

    Raúl Montes-de-Oca

    2013-01-01

    Full Text Available From the classical point of view, it is important to determine if in a Markov decision process (MDP, besides their existence, the uniqueness of the optimal policies is guaranteed. It is well known that uniqueness does not always hold in optimization problems (for instance, in linear programming. On the other hand, in such problems it is possible for a slight perturbation of the functional cost to restore the uniqueness. In this paper, it is proved that the value functions of an MDP and its cost perturbed version stay close, under adequate conditions, which in some sense is a priority. We are interested in the stability of Markov decision processes with respect to the perturbations of the cost-as-you-go function.

  13. Digital signal processing for fiber nonlinearities [Invited

    DEFF Research Database (Denmark)

    Cartledge, John C.; Guiomar, Fernando P.; Kschischang, Frank R.

    2017-01-01

    This paper reviews digital signal processing techniques that compensate, mitigate, and exploit fiber nonlinearities in coherent optical fiber transmission systems......This paper reviews digital signal processing techniques that compensate, mitigate, and exploit fiber nonlinearities in coherent optical fiber transmission systems...

  14. The Langevin Approach: An R Package for Modeling Markov Processes

    Directory of Open Access Journals (Sweden)

    Philip Rinn

    2016-08-01

    Full Text Available We describe an 'R' package developed by the research group 'Turbulence, Wind energy' 'and Stochastics' (TWiSt at the Carl von Ossietzky University of Oldenburg, which extracts the (stochastic evolution equation underlying a set of data or measurements. The method can be directly applied to data sets with one or two stochastic variables. Examples for the one-dimensional and two-dimensional cases are provided. This framework is valid under a small set of conditions which are explicitly presented and which imply simple preliminary test procedures to the data. For Markovian processes involving Gaussian white noise, a stochastic differential equation is derived straightforwardly from the time series and captures the full dynamical properties of the underlying process. Still, even in the case such conditions are not fulfilled, there are alternative versions of this method which we discuss briefly and provide the user with the necessary bibliography.

  15. The Langevin Approach: An R Package for Modeling Markov Processes

    CERN Document Server

    Rinn, Philip; Wächter, Matthias; Peinke, Joachim

    2016-01-01

    We describe an R package developed by the research group Turbulence, Wind energy and Stochastics (TWiSt) at the Carl von Ossietzky University of Oldenburg, which extracts the (stochastic) evolution equation underlying a set of data or measurements. The method can be directly applied to data sets with one or two stochastic variables. Examples for the one-dimensional and two-dimensional cases are provided. This framework is valid under a small set of conditions which are explicitly presented and which imply simple preliminary test procedures to the data. For Markovian processes involving Gaussian white noise, a stochastic differential equation is derived straightforwardly from the time series and captures the full dynamical properties of the underlying process. Still, even in the case such conditions are not fulfilled, there are alternative versions of this method which we discuss briefly and provide the user with the necessary bibliography.

  16. Existence of the optimal measurable coupling and ergodicity for Markov processes

    Institute of Scientific and Technical Information of China (English)

    张绍义

    1999-01-01

    The existence theorem of the optimal measurable coupling of two probability kernels on a complete separable metric measurable space is proved. Then by this theorem, a general ergodicity theorem for Markov processes is obtained. And as an immediate application to particle systems the uniqueness theorem of the stationary distribution is supplemented, i.e. the uniqueness theorem also implies the existence of the stationary distribution.

  17. Assistive system for people with Apraxia using a Markov decision process.

    Science.gov (United States)

    Jean-Baptiste, Emilie M D; Russell, Martin; Rothstein, Pia

    2014-01-01

    CogWatch is an assistive system to re-train stroke survivors suffering from Apraxia or Action Disorganization Syndrome (AADS) to complete activities of daily living (ADLs). This paper describes the approach to real-time planning based on a Markov Decision Process (MDP), and demonstrates its ability to improve task's performance via user simulation. The paper concludes with a discussion of the remaining challenges and future enhancements.

  18. Invariant Measure for the Markov Process Corresponding to a PDE System

    Institute of Scientific and Technical Information of China (English)

    Fu Bao XI

    2005-01-01

    In this paper, we consider the Markov process (X∈(t), Z∈(t)) corresponding to a weakly coupled elliptic PDE system with a small parameter ∈> 0. We first prove that (X∈(t), Z∈(t)) has the Feller continuity by the coupling method, and then prove that (X∈(t), Z∈(t)) has an invariant measure the small parameter ∈ tends to zero.

  19. «Concurrency» in M-L-Parallel Semi-Markov Process

    Directory of Open Access Journals (Sweden)

    Larkin Eugene

    2017-01-01

    Full Text Available This article investigates the functioning of a swarm of robots, each of which receives instructions from the external human operator and autonomously executes them. An abstract model of functioning of a robot, a group of robots and multiple groups of robots was obtained using the notion of semi-Markov process. The concepts of aggregated initial and aggregated absorbing states were introduced. Correspondences for calculation of time parameters of concurrency were obtained.

  20. Randomized Search Methods for Solving Markov Decision Processes and Global Optimization

    Science.gov (United States)

    2006-01-01

    over relaxation (SOR) method ([81]). Puterman and Shin [62] proposed a modified policy iteration algorithm, which takes the basic form of PI, with the...99018) (1999). [61] Pintér, J. D., Global Optimization in Action, Kluwer Academic Publisher, The Netherlands, 1996. [62] Puterman , M. L. and Shin, M. C...Modified policy iteration algorithms for dis- counted Markov decision processes,” Management Science, 24, 1127–1137 (1978). [63] Puterman , M. L

  1. Phase-Type Approximations for Wear Processes in A Semi-Markov Environment

    Science.gov (United States)

    2004-03-01

    identically distributed exponential random variables, is equivalent to the absorption time of an underlying k-state Markov process. As noted by Perros ...the Coxian distribution is that it can exactly represent any distribution having a rational Laplace transform [23]. Moreover, Perros [23] gives the...Performance Evaluation (TOOLS 2003), 200-217. 23. Perros , H. (1994). Queueing Networks with Blocking. Oxford University Press, New York. 24. Ro, C.W

  2. An Overview of Markov Chain Methods for the Study of Stage-Sequential Developmental Processes

    Science.gov (United States)

    Kapland, David

    2008-01-01

    This article presents an overview of quantitative methodologies for the study of stage-sequential development based on extensions of Markov chain modeling. Four methods are presented that exemplify the flexibility of this approach: the manifest Markov model, the latent Markov model, latent transition analysis, and the mixture latent Markov model.…

  3. Enantiodromic effective generators of a Markov jump process with Gallavotti-Cohen symmetry

    Science.gov (United States)

    Terohid, S. A. A.; Torkaman, P.; Jafarpour, F. H.

    2016-11-01

    This paper deals with the properties of the stochastic generators of the effective (driven) processes associated with atypical values of transition-dependent time-integrated currents with Gallavotti-Cohen symmetry in Markov jump processes. Exploiting the concept of biased ensemble of trajectories by introducing a biasing field s , we show that the stochastic generators of the effective processes associated with the biasing fields s and E -s are enantiodromic with respect to each other where E is the conjugated field to the current. We illustrate our findings by considering an exactly solvable creation-annihilation process of classical particles with nearest-neighbor interactions defined on a one-dimensional lattice.

  4. Variance-penalized Markov decision processes: dynamic programming and reinforcement learning techniques

    Science.gov (United States)

    Gosavi, Abhijit

    2014-08-01

    In control systems theory, the Markov decision process (MDP) is a widely used optimization model involving selection of the optimal action in each state visited by a discrete-event system driven by Markov chains. The classical MDP model is suitable for an agent/decision-maker interested in maximizing expected revenues, but does not account for minimizing variability in the revenues. An MDP model in which the agent can maximize the revenues while simultaneously controlling the variance in the revenues is proposed. This work is rooted in machine learning/neural network concepts, where updating is based on system feedback and step sizes. First, a Bellman equation for the problem is proposed. Thereafter, convergent dynamic programming and reinforcement learning techniques for solving the MDP are provided along with encouraging numerical results on a small MDP and a preventive maintenance problem.

  5. Availability Control for Means of Transport in Decisive Semi-Markov Models of Exploitation Process

    Science.gov (United States)

    Migawa, Klaudiusz

    2012-12-01

    The issues presented in this research paper refer to problems connected with the control process for exploitation implemented in the complex systems of exploitation for technical objects. The article presents the description of the method concerning the control availability for technical objects (means of transport) on the basis of the mathematical model of the exploitation process with the implementation of the decisive processes by semi-Markov. The presented method means focused on the preparing the decisive for the exploitation process for technical objects (semi-Markov model) and after that specifying the best control strategy (optimal strategy) from among possible decisive variants in accordance with the approved criterion (criteria) of the activity evaluation of the system of exploitation for technical objects. In the presented method specifying the optimal strategy for control availability in the technical objects means a choice of a sequence of control decisions made in individual states of modelled exploitation process for which the function being a criterion of evaluation reaches the extreme value. In order to choose the optimal control strategy the implementation of the genetic algorithm was chosen. The opinions were presented on the example of the exploitation process of the means of transport implemented in the real system of the bus municipal transport. The model of the exploitation process for the means of transports was prepared on the basis of the results implemented in the real transport system. The mathematical model of the exploitation process was built taking into consideration the fact that the model of the process constitutes the homogenous semi-Markov process.

  6. A reward semi-Markov process with memory for wind speed modeling

    Science.gov (United States)

    Petroni, F.; D'Amico, G.; Prattico, F.

    2012-04-01

    -order Markov chain with different number of states, and Weibull distribution. All this model use Markov chains to generate synthetic wind speed time series but the search for a better model is still open. Approaching this issue, we applied new models which are generalization of Markov models. More precisely we applied semi-Markov models to generate synthetic wind speed time series. The primary goal of this analysis is the study of the time history of the wind in order to assess its reliability as a source of power and to determine the associated storage levels required. In order to assess this issue we use a probabilistic model based on indexed semi-Markov process [4] to which a reward structure is attached. Our model is used to calculate the expected energy produced by a given turbine and its variability expressed by the variance of the process. Our results can be used to compare different wind farms based on their reward and also on the risk of missed production due to the intrinsic variability of the wind speed process. The model is used to generate synthetic time series for wind speed by means of Monte Carlo simulations and backtesting procedure is used to compare results on first and second oder moments of rewards between real and synthetic data. [1] A. Shamshad, M.A. Bawadi, W.M.W. Wan Hussin, T.A. Majid, S.A.M. Sanusi, First and second order Markov chain models for synthetic gen- eration of wind speed time series, Energy 30 (2005) 693-708. [2] H. Nfaoui, H. Essiarab, A.A.M. Sayigh, A stochastic Markov chain model for simulating wind speed time series at Tangiers, Morocco, Re- newable Energy 29 (2004) 1407-1418. [3] F. Youcef Ettoumi, H. Sauvageot, A.-E.-H. Adane, Statistical bivariate modeling of wind using first-order Markov chain and Weibull distribu- tion, Renewable Energy 28 (2003) 1787-1802. [4]F. Petroni, G. D'Amico, F. Prattico, Indexed semi-Markov process for wind speed modeling. To be submitted.

  7. Wind speed modeled as a semi-Markov process with memory

    CERN Document Server

    D'Amico, Guglielmo; Prattico, Flavio

    2012-01-01

    The increasing interest in renewable energy, particularly in wind, has given rise to the necessity of accurate models for the generation of good synthetic wind speed data. Markov chains are often used with this purpose but better models are needed to reproduce the statistical properties of wind speed data. In a previous paper we showed that semi-Markov processes are more appropriate for this purpose but to reach an accurate reproduction of real data features high order model should be used. In this work we introduce an indexed semi-Markov process that is able to fit real data. We downloaded a database, freely available from the web, in which are included wind speed data taken from L.S.I. -Lastem station (Italy) and sampled every 10 minutes. We then generate synthetic time series for wind speed by means of Monte Carlo simulations. The time lagged autocorrelation is then used to compare statistical properties of the proposed model with those of real data and also with a synthetic time series generated though a ...

  8. Determination of an Unknown Radiation Term in a Nonlinear Inverse Problem using Simulation of Markov Chains

    Directory of Open Access Journals (Sweden)

    Morteza Ebrahimi

    2012-01-01

    Full Text Available The purpose of the present study is to provide a fast and accurate algorithm for identifying the medium temperature and the unknown radiation term from an overspecified condition on the boundary in an inverse problem of linear heat equation with nonlinear boundary condition. The design of the paper is to employ Taylor’s series expansion for linearize nonlinear term and then finite-difference approximation to discretize the problem domain. Owing to the application of the finite difference scheme, a large sparse system of linear algebraic equations is obtained. An approach of Monte Carlo method is employed to solve the linear system and estimate unknown radiation term. The Monte Carlo optimization is adopted to modify the estimated values. Results show that a good estimation on the radiation term can be obtained within a couple of minutes CPU time at pentium IV-2.4 GHz PC.

  9. The discovery of processing stages: analyzing EEG data with hidden semi-Markov models.

    Science.gov (United States)

    Borst, Jelmer P; Anderson, John R

    2015-03-01

    In this paper we propose a new method for identifying processing stages in human information processing. Since the 1860s scientists have used different methods to identify processing stages, usually based on reaction time (RT) differences between conditions. To overcome the limitations of RT-based methods we used hidden semi-Markov models (HSMMs) to analyze EEG data. This HSMM-EEG methodology can identify stages of processing and how they vary with experimental condition. By combining this information with the brain signatures of the identified stages one can infer their function, and deduce underlying cognitive processes. To demonstrate the method we applied it to an associative recognition task. The stage-discovery method indicated that three major processes play a role in associative recognition: a familiarity process, an associative retrieval process, and a decision process. We conclude that the new stage-discovery method can provide valuable insight into human information processing.

  10. Input saturation in nonlinear multivariable processes resolved by nonlinear decoupling

    Directory of Open Access Journals (Sweden)

    Jens G. Balchen

    1995-04-01

    Full Text Available A new method is presented for the resolution of the problem of input saturation in nonlinear multivariable process control by means of elementary nonlinear decoupling (END. Input saturation can have serious consequences particularly in multivariable control because it may lead to very undesirable system behaviour and quite often system instability. Many authors have searched for systematic techniques for designing multivariable control systems in which saturation may occur in any of the control variables (inputs, manipulated variables. No generally accepted method seems to have been presented so far which gives a solution in closed form. The method of elementary nonlinear decoupling (END can be applied directly to the case of saturation control variables by deriving as many control strategies as there are combinations of saturating control variables. The method is demonstrated by the multivariable control of a simulated Fluidized Catalytic Cracker (FCC with very convincing results.

  11. A hierarchical Markov decision process modeling feeding and marketing decisions of growing pigs

    DEFF Research Database (Denmark)

    Pourmoayed, Reza; Nielsen, Lars Relund; Kristensen, Anders Ringgaard

    2016-01-01

    Feeding is the most important cost in the production of growing pigs and has a direct impact on the marketing decisions, growth and the final quality of the meat. In this paper, we address the sequential decision problem of when to change the feed-mix within a finisher pig pen and when to pick pigs...... for marketing. We formulate a hierarchical Markov decision process with three levels representing the decision process. The model considers decisions related to feeding and marketing and finds the optimal decision given the current state of the pen. The state of the system is based on information from on...

  12. A statistical property of multiagent learning based on Markov decision process.

    Science.gov (United States)

    Iwata, Kazunori; Ikeda, Kazushi; Sakai, Hideaki

    2006-07-01

    We exhibit an important property called the asymptotic equipartition property (AEP) on empirical sequences in an ergodic multiagent Markov decision process (MDP). Using the AEP which facilitates the analysis of multiagent learning, we give a statistical property of multiagent learning, such as reinforcement learning (RL), near the end of the learning process. We examine the effect of the conditions among the agents on the achievement of a cooperative policy in three different cases: blind, visible, and communicable. Also, we derive a bound on the speed with which the empirical sequence converges to the best sequence in probability, so that the multiagent learning yields the best cooperative result.

  13. Multivariate Markov processes for stochastic systems with delays: application to the stochastic Gompertz model with delay.

    Science.gov (United States)

    Frank, T D

    2002-07-01

    Using the method of steps, we describe stochastic processes with delays in terms of Markov diffusion processes. Thus, multivariate Langevin equations and Fokker-Planck equations are derived for stochastic delay differential equations. Natural, periodic, and reflective boundary conditions are discussed. Both Ito and Stratonovich calculus are used. In particular, our Fokker-Planck approach recovers the generalized delay Fokker-Planck equation proposed by Guillouzic et al. The results obtained are applied to a model for population growth: the Gompertz model with delay and multiplicative white noise.

  14. Spectral Analysis of Multi-dimensional Self-similar Markov Processes

    CERN Document Server

    Modarresi, N

    2009-01-01

    In this paper we consider a wide sense discrete scale invariant process with scale $l>1$. We consider to have $T$ samples at each scale, and choose $\\alpha$ by the equality $l=\\alpha^T$. Our special scheme of sampling is to choose our samples at discrete points $\\alpha^k, k\\in W$. So we provide a discrete time wide sense scale invariant(DT-SI) process. We find the spectral representation of the covariance function of such DT-SI process. By providing harmonic like representation of multi-dimensional self-similar processes, spectral density function of them are presented. We also consider a discrete time scale invariance Markov(DT-SIM) process with the above scheme of sampling at points $\\alpha^k, k\\in {\\bf W}$ and show that the spectral density matrix of DT-SIM process and its associated $T$-dimensional self-similar Markov process is fully specified by $\\{R_{j}^H(1),R_{j}^H(0),j=0, 1, ..., T-1\\}$ where $R_{j}^H(\\tau)=\\mathrm{Cov}\\big(X(\\alpha^{j+\\tau}),X(\\alpha^j)\\big)$

  15. Existence,uniqueness and ergodicity of Markov branching processes with immigration and instantaneous resurrection

    Institute of Scientific and Technical Information of China (English)

    2008-01-01

    We consider a modified Markov branching process incorporating with both state-independent immigration and instantaneous resurrection.The existence criterion of the process is firstly considered.We prove that if the sum of the resurrection rates is finite,then there does not exist any process.An existence criterion is then established when the sum of the resurrection rates is infinite.Some equivalent criteria,possessing the advantage of being easily checked,are obtained for the latter case.The uniqueness criterion for such process is also investigated.We prove that although there exist infinitely many of them,there always exists a unique honest process for a given q-matrix.This unique honest process is then constructed.The ergodicity property of this honest process is analysed in detail.We prove that this honest process is always ergodic and the explicit expression for the equilibrium distribution is established.

  16. Existence,uniqueness and ergodicity of Markov branching processes with immigration and instantaneous resurrection

    Institute of Scientific and Technical Information of China (English)

    LI JunPing; CHEN AnYue

    2008-01-01

    We consider a modified Markov branching process incorporating with both stateindependent immigration and instantaneous resurrection. The existence criterion of the process is firstly considered. We prove that if the sum of the resurrection rates is finite, then there does not exist any process. An existence criterion is then established when the sum of the resurrection rates is infinite.Some equivalent criteria, possessing the advantage of being easily checked, are obtained for the latter case. The uniqueness criterion for such process is also investigated. We prove that although there exist infinitely many of them, there always exists a unique honest process for a given q-matrix. This unique honest process is then constructed. The ergodicity property of this honest process is analysed in detail.We prove that this honest process is always ergodic and the explicit expression for the equilibrium distribution is established.

  17. Adiabatic reduction of a model of stochastic gene expression with jump Markov process.

    Science.gov (United States)

    Yvinec, Romain; Zhuge, Changjing; Lei, Jinzhi; Mackey, Michael C

    2014-04-01

    This paper considers adiabatic reduction in a model of stochastic gene expression with bursting transcription considered as a jump Markov process. In this model, the process of gene expression with auto-regulation is described by fast/slow dynamics. The production of mRNA is assumed to follow a compound Poisson process occurring at a rate depending on protein levels (the phenomena called bursting in molecular biology) and the production of protein is a linear function of mRNA numbers. When the dynamics of mRNA is assumed to be a fast process (due to faster mRNA degradation than that of protein) we prove that, with appropriate scalings in the burst rate, jump size or translational rate, the bursting phenomena can be transmitted to the slow variable. We show that, depending on the scaling, the reduced equation is either a stochastic differential equation with a jump Poisson process or a deterministic ordinary differential equation. These results are significant because adiabatic reduction techniques seem to have not been rigorously justified for a stochastic differential system containing a jump Markov process. We expect that the results can be generalized to adiabatic methods in more general stochastic hybrid systems.

  18. Sensor Network Design for Nonlinear Processes

    Institute of Scientific and Technical Information of China (English)

    李博; 陈丙珍

    2003-01-01

    This paper presents a method to design a cost-optimal nonredundant sensor network to observe all variables in a general nonlinear process. A mixed integer linear programming model was used to minimize the cost with data classification to check the observability of all unmeasured variables. This work is a starting point for designing sensor networks for general nonlinear processes based on various criteria, such as reliability and accuracy.

  19. Application of Markov process modelling to health status switching behaviour of infants.

    Science.gov (United States)

    Biritwum, R B; Odoom, S I

    1995-02-01

    This study is an attempt to apply Markov process modelling to health status switching behaviour of infants. The data for the study consist of monthly records of diagnosed illnesses for 1152 children, each observed from the month of first contact with Kasangati Health Centre, Kampala, Uganda, until age 18 months. Only two states of health are considered in the study, a 'Health' state, denoted by W: (for Well), and an 'Illness' state denoted by S: (for Sick). The data are thus reduced to monthly records (W or S) of the states of health of the study sample. The simplest model of dependence of current health state on the past is one that links the current state to the immediately preceding month only; that is a Markov model. The starting point of this study was therefore to determine the proportions of children making the transitions W-->W, W-->S, S-->W, S-->S, from one month to the next, for each month from birth (month 0) to 18 months of age (month 18). These were used as estimates of the probabilities of making these transitions for each month from birth. This paper discusses the main features emerging from the study of these transition probabilities. In the first 5 months after birth, the probabilities of making the transitions W-->W, W-->S, S-->W, S-->S from one month to the next, showed some dependence on the age of the child. From the sixth month on, however, the dependence on age seemed to wear off. The transition probabilities remained the same from then on, suggesting that the switching pattern between health states behaves, eventually, like a time-homogeneous Markov Chain. This time-homogeneous chain attained a steady state distribution at about 12 months from birth. The study has shown that the transitions between Health and Illness for infants, from month to month, can be modelled by a Markov Chain for which the (single-step) transition probabilities are generally time-dependent or age-dependent. After the first few months of life the dependence on age may

  20. Central Limit Theorem for Nonlinear Hawkes Processes

    CERN Document Server

    Zhu, Lingjiong

    2012-01-01

    Hawkes process is a self-exciting point process with clustering effect whose jump rate depends on its entire past history. It has wide applications in neuroscience, finance and many other fields. Linear Hawkes process has an immigration-birth representation and can be computed more or less explicitly. It has been extensively studied in the past and the limit theorems are well understood. On the contrary, nonlinear Hawkes process lacks the immigration-birth representation and is much harder to analyze. In this paper, we obtain a functional central limit theorem for nonlinear Hawkes process.

  1. Viability, Invariance and Reachability for Controlled Piecewise Deterministic Markov Processes Associated to Gene Networks

    CERN Document Server

    Goreac, D

    2010-01-01

    We aim at characterizing viability, invariance and some reachability properties of controlled piecewise deterministic Markov processes (PDMPs). Using analytical methods from the theory of viscosity solutions, we establish criteria for viability and invariance in terms of the first order normal cone. We also investigate reachability of arbitrary open sets. The method is based on viscosity techniques and duality for some associated linearized problem. The theoretical results are applied to general On/Off systems, Cook's model for haploinssuficiency, and a stochastic model for bacteriophage lambda.

  2. Markov chain Monte Carlo methods for state-space models with point process observations.

    Science.gov (United States)

    Yuan, Ke; Girolami, Mark; Niranjan, Mahesan

    2012-06-01

    This letter considers how a number of modern Markov chain Monte Carlo (MCMC) methods can be applied for parameter estimation and inference in state-space models with point process observations. We quantified the efficiencies of these MCMC methods on synthetic data, and our results suggest that the Reimannian manifold Hamiltonian Monte Carlo method offers the best performance. We further compared such a method with a previously tested variational Bayes method on two experimental data sets. Results indicate similar performance on the large data sets and superior performance on small ones. The work offers an extensive suite of MCMC algorithms evaluated on an important class of models for physiological signal analysis.

  3. A Stable Clock Error Model Using Coupled First and Second Order Gauss-Markov Processes

    Science.gov (United States)

    Carpenter, Russell; Lee, Taesul

    2008-01-01

    Long data outages may occur in applications of global navigation satellite system technology to orbit determination for missions that spend significant fractions of their orbits above the navigation satellite constellation(s). Current clock error models based on the random walk idealization may not be suitable in these circumstances, since the covariance of the clock errors may become large enough to overflow flight computer arithmetic. A model that is stable, but which approximates the existing models over short time horizons is desirable. A coupled first- and second-order Gauss-Markov process is such a model.

  4. Effective degree Markov-chain approach for discrete-time epidemic processes on uncorrelated networks.

    Science.gov (United States)

    Cai, Chao-Ran; Wu, Zhi-Xi; Guan, Jian-Yue

    2014-11-01

    Recently, Gómez et al. proposed a microscopic Markov-chain approach (MMCA) [S. Gómez, J. Gómez-Gardeñes, Y. Moreno, and A. Arenas, Phys. Rev. E 84, 036105 (2011)PLEEE81539-375510.1103/PhysRevE.84.036105] to the discrete-time susceptible-infected-susceptible (SIS) epidemic process and found that the epidemic prevalence obtained by this approach agrees well with that by simulations. However, we found that the approach cannot be straightforwardly extended to a susceptible-infected-recovered (SIR) epidemic process (due to its irreversible property), and the epidemic prevalences obtained by MMCA and Monte Carlo simulations do not match well when the infection probability is just slightly above the epidemic threshold. In this contribution we extend the effective degree Markov-chain approach, proposed for analyzing continuous-time epidemic processes [J. Lindquist, J. Ma, P. Driessche, and F. Willeboordse, J. Math. Biol. 62, 143 (2011)JMBLAJ0303-681210.1007/s00285-010-0331-2], to address discrete-time binary-state (SIS) or three-state (SIR) epidemic processes on uncorrelated complex networks. It is shown that the final epidemic size as well as the time series of infected individuals obtained from this approach agree very well with those by Monte Carlo simulations. Our results are robust to the change of different parameters, including the total population size, the infection probability, the recovery probability, the average degree, and the degree distribution of the underlying networks.

  5. Broadband Nonlinear Signal Processing in Silicon Nanowires

    DEFF Research Database (Denmark)

    Yvind, Kresten; Pu, Minhao; Hvam, Jørn Märcher;

    The fast non-linearity of silicon allows Tbit/s optical signal processing. By choosing suitable dimensions of silicon nanowires their dispersion can be tailored to ensure a high nonlinearity at power levels low enough to avoid significant two-photon abso We have fabricated low insertion and propa......The fast non-linearity of silicon allows Tbit/s optical signal processing. By choosing suitable dimensions of silicon nanowires their dispersion can be tailored to ensure a high nonlinearity at power levels low enough to avoid significant two-photon abso We have fabricated low insertion...... and propagation loss silicon nanowires and use them to demonstrate the broadband capabilities of silicon....

  6. A patterned and un-patterned minefield detection in cluttered environments using Markov marked point process

    Science.gov (United States)

    Trang, Anh; Agarwal, Sanjeev; Regalia, Phillip; Broach, Thomas; Smith, Thomas

    2007-04-01

    A typical minefield detection approach is based on a sequential processing employing mine detection and false alarm rejection followed by minefield detection. The current approach does not work robustly under different backgrounds and environment conditions because target signature changes with time and its performance degrades in the presence of high density of false alarms. The aim of this research will be to advance the state of the art in detection of both patterned and unpatterned minefield in high clutter environments. The proposed method seeks to combine false alarm rejection module and the minefield detection module of the current architecture by spatial-spectral clustering and inference module using a Markov Marked Point Process formulation. The approach simultaneously exploits the feature characteristics of the target signature and spatial distribution of the targets in the interrogation region. The method is based on the premise that most minefields can be characterized by some type of distinctive spatial distribution of "similar" looking mine targets. The minefield detection problem is formulated as a Markov Marked Point Process (MMPP) where the set of possible mine targets is divided into a possibly overlapping mixture of targets. The likelihood of the minefield depends simultaneously on feature characteristics of the target and their spatial distribution. A framework using "Belief Propagation" is developed to solve the minefield inference problem based on MMPP. Preliminary investigation using simulated data shows the efficacy of the approach.

  7. On-line monitoring of pharmaceutical production processes using Hidden Markov Model.

    Science.gov (United States)

    Zhang, Hui; Jiang, Zhuangde; Pi, J Y; Xu, H K; Du, R

    2009-04-01

    This article presents a new method for on-line monitoring of pharmaceutical production process, especially the powder blending process. The new method consists of two parts: extracting features from the Near Infrared (NIR) spectroscopy signals and recognizing patterns from the features. Features are extracted from spectra by using Partial Least Squares method (PLS). The pattern recognition is done by using Hidden Markov Model (HMM). A series of experiments are conducted to evaluate the effectiveness of this new method. In the experiments, wheat powder and corn powder are blended together at a set concentration. The proposed method can effectively detect the blending uniformity (the success rate is 99.6%). In comparison to the conventional Moving Block of Standard Deviation (MBSD), the proposed method has a number of advantages, including higher reliability, higher robustness and more transparent decision making. It can be used for effective on-line monitoring of pharmaceutical production processes.

  8. On the Limiting Markov Process of Energy Exchanges in a Rarely Interacting Ball-Piston Gas

    Science.gov (United States)

    Bálint, Péter; Gilbert, Thomas; Nándori, Péter; Szász, Domokos; Tóth, Imre Péter

    2017-02-01

    We analyse the process of energy exchanges generated by the elastic collisions between a point-particle, confined to a two-dimensional cell with convex boundaries, and a `piston', i.e. a line-segment, which moves back and forth along a one-dimensional interval partially intersecting the cell. This model can be considered as the elementary building block of a spatially extended high-dimensional billiard modeling heat transport in a class of hybrid materials exhibiting the kinetics of gases and spatial structure of solids. Using heuristic arguments and numerical analysis, we argue that, in a regime of rare interactions, the billiard process converges to a Markov jump process for the energy exchanges and obtain the expression of its generator.

  9. Exploring the WTI crude oil price bubble process using the Markov regime switching model

    Science.gov (United States)

    Zhang, Yue-Jun; Wang, Jing

    2015-03-01

    The sharp volatility of West Texas Intermediate (WTI) crude oil price in the past decade triggers us to investigate the price bubbles and their evolving process. Empirical results indicate that the fundamental price of WTI crude oil appears relatively more stable than that of the market-trading price, which verifies the existence of oil price bubbles during the sample period. Besides, by allowing the WTI crude oil price bubble process to switch between two states (regimes) according to a first-order Markov chain, we are able to statistically discriminate upheaval from stable states in the crude oil price bubble process; and in most of time, the stable state dominates the WTI crude oil price bubbles while the upheaval state usually proves short-lived and accompanies unexpected market events.

  10. On-line Fault Diagnosis in Industrial Processes Using Variable Moving Window and Hidden Markov Model

    Institute of Scientific and Technical Information of China (English)

    周韶园; 谢磊; 王树青

    2005-01-01

    An integrated framework is presented to represent and classify process data for on-line identifying abnormal operating conditions. It is based on pattern recognition principles and consists of a feature extraction step, by which wavelet transform and principal component analysis are used to capture the inherent characteristics from process measurements, followed by a similarity assessment step using hidden Markov model (HMM) for pattern comparison. In most previous cases, a fixed-length moving window was employed to track dynamic data, and often failed to capture enough information for each fault and sometimes even deteriorated the diagnostic performance. A variable moving window, the length of which is modified with time, is introduced in this paper and case studies on the Tennessee Eastman process illustrate the potential of the proposed method.

  11. An implicit finite volume scheme for a scalar hyperbolic problem with measure data related to piecewise deterministic Markov processes

    Science.gov (United States)

    Eymard, Robert; Mercier, Sophie; Prignet, Alain

    2008-12-01

    We are interested here in the numerical approximation of a family of probability measures, solution of the Chapman-Kolmogorov equation associated to some non-diffusion Markov process with uncountable state space. Such an equation contains a transport term and another term, which implies redistribution of the probability mass on the whole space. An implicit finite volume scheme is proposed, which is intermediate between an upstream weighting scheme and a modified Lax-Friedrichs one. Due to the seemingly unusual probability framework, a new weak bounded variation inequality had to be developed, in order to prove the convergence of the discretised transport term. Such an inequality may be used in other contexts, such as for the study of finite volume approximations of scalar linear or nonlinear hyperbolic equations with initial data in L1. Also, due to the redistribution term, the tightness of the family of approximate probability measures had to be proven. Numerical examples are provided, showing the efficiency of the implicit finite volume scheme and its potentiality to be helpful in an industrial reliability context.

  12. Ultrafast Nonlinear Signal Processing in Silicon Waveguides

    DEFF Research Database (Denmark)

    Oxenløwe, Leif Katsuo; Mulvad, Hans Christian Hansen; Hu, Hao;

    2012-01-01

    We describe recent demonstrations of exploiting highly nonlinear silicon waveguides for ultrafast optical signal processing. We describe wavelength conversion and serial-to-parallel conversion of 640 Gbit/s data signals and 1.28 Tbit/s demultiplexing and all-optical sampling.......We describe recent demonstrations of exploiting highly nonlinear silicon waveguides for ultrafast optical signal processing. We describe wavelength conversion and serial-to-parallel conversion of 640 Gbit/s data signals and 1.28 Tbit/s demultiplexing and all-optical sampling....

  13. Discounted continuous-time constrained Markov decision processes in Polish spaces

    CERN Document Server

    Guo, Xianping; 10.1214/10-AAP749

    2012-01-01

    This paper is devoted to studying constrained continuous-time Markov decision processes (MDPs) in the class of randomized policies depending on state histories. The transition rates may be unbounded, the reward and costs are admitted to be unbounded from above and from below, and the state and action spaces are Polish spaces. The optimality criterion to be maximized is the expected discounted rewards, and the constraints can be imposed on the expected discounted costs. First, we give conditions for the nonexplosion of underlying processes and the finiteness of the expected discounted rewards/costs. Second, using a technique of occupation measures, we prove that the constrained optimality of continuous-time MDPs can be transformed to an equivalent (optimality) problem over a class of probability measures. Based on the equivalent problem and a so-called $\\bar{w}$-weak convergence of probability measures developed in this paper, we show the existence of a constrained optimal policy. Third, by providing a linear ...

  14. First Passage Moments of Finite-State Semi-Markov Processes

    Energy Technology Data Exchange (ETDEWEB)

    Warr, Richard [Los Alamos National Lab. (LANL), Los Alamos, NM (United States); Cordeiro, James [Air Force Research Lab. (AFRL), Wright-Patterson AFB, OH (United States)

    2014-03-31

    In this paper, we discuss the computation of first-passage moments of a regular time-homogeneous semi-Markov process (SMP) with a finite state space to certain of its states that possess the property of universal accessibility (UA). A UA state is one which is accessible from any other state of the SMP, but which may or may not connect back to one or more other states. An important characteristic of UA is that it is the state-level version of the oft-invoked process-level property of irreducibility. We adapt existing results for irreducible SMPs to the derivation of an analytical matrix expression for the first passage moments to a single UA state of the SMP. In addition, consistent point estimators for these first passage moments, together with relevant R code, are provided.

  15. Modeling treatment of ischemic heart disease with partially observable Markov decision processes.

    Science.gov (United States)

    Hauskrecht, M; Fraser, H

    1998-01-01

    Diagnosis of a disease and its treatment are not separate, one-shot activities. Instead they are very often dependent and interleaved over time, mostly due to uncertainty about the underlying disease, uncertainty associated with the response of a patient to the treatment and varying cost of different diagnostic (investigative) and treatment procedures. The framework of Partially observable Markov decision processes (POMDPs) developed and used in operations research, control theory and artificial intelligence communities is particularly suitable for modeling such a complex decision process. In the paper, we show how the POMDP framework could be used to model and solve the problem of the management of patients with ischemic heart disease, and point out modeling advantages of the framework over standard decision formalisms.

  16. Planning treatment of ischemic heart disease with partially observable Markov decision processes.

    Science.gov (United States)

    Hauskrecht, M; Fraser, H

    2000-03-01

    Diagnosis of a disease and its treatment are not separate, one-shot activities. Instead, they are very often dependent and interleaved over time. This is mostly due to uncertainty about the underlying disease, uncertainty associated with the response of a patient to the treatment and varying cost of different diagnostic (investigative) and treatment procedures. The framework of partially observable Markov decision processes (POMDPs) developed and used in the operations research, control theory and artificial intelligence communities is particularly suitable for modeling such a complex decision process. In this paper, we show how the POMDP framework can be used to model and solve the problem of the management of patients with ischemic heart disease (IHD), and demonstrate the modeling advantages of the framework over standard decision formalisms.

  17. A non-Markov ratchet model of molecular motors: processive movement of single-headed kinesin KIF1A

    Institute of Scientific and Technical Information of China (English)

    Xie Ping; Dou Shuo-Xing; Wang Peng-Ye

    2006-01-01

    A fluctuating ratchet model of non-Markov process is presented to describe the processive movement of molecular motors of single-headed kinesin KIF1A, where the fluctuation perturbation to the local potential is introduced and the detailed ATPase pathway of the motor is included. The theoretical results show good quantitative agreement with the previous experimental ones.

  18. An Online Policy Gradient Algorithm for Markov Decision Processes with Continuous States and Actions.

    Science.gov (United States)

    Ma, Yao; Zhao, Tingting; Hatano, Kohei; Sugiyama, Masashi

    2016-03-01

    We consider the learning problem under an online Markov decision process (MDP) aimed at learning the time-dependent decision-making policy of an agent that minimizes the regret-the difference from the best fixed policy. The difficulty of online MDP learning is that the reward function changes over time. In this letter, we show that a simple online policy gradient algorithm achieves regret O(√T) for T steps under a certain concavity assumption and O(log T) under a strong concavity assumption. To the best of our knowledge, this is the first work to present an online MDP algorithm that can handle continuous state, action, and parameter spaces with guarantee. We also illustrate the behavior of the proposed online policy gradient method through experiments.

  19. On the Controller Synthesis for Markov Decision Process of Conflict Tolerant Specification

    Science.gov (United States)

    Zhang, Junhua; Huang, Zhiqiu; Cao, Zining

    For an embedded control system, different requirements often need be satisfied at same time, and some of them make the system to act conflicted. Conflict tolerant specification is provided to denote this situation. In such a system, there often exist probabilistic and non-deterministic behaviors. We use Markov Decision Process (MDP) to denote these features. We study the controller synthesis for MDP over conflict tolerant specification. We extend PCTL star by adding past operator to denote the conflict tolerant specification succinctly. We use CT-PLTL to denote conflicted actions and PCTL to denote the specification for probability demand. We first synthesize a controller on a base system over CT-PLTL and then use it to prune the corresponding MDP of the system model. We use the resulting sub-MDP as the model to further synthesis a controller over PCTL. The whole controller for MDP is a conjunction of the two controllers obtained.

  20. Average Sample-path Optimality for Continuous-time Markov Decision Processes in Polish Spaces

    Institute of Scientific and Technical Information of China (English)

    Quan-xin ZHU

    2011-01-01

    In this paper we study the average sample-path cost (ASPC) problem for continuous-time Markov decision processes in Polish spaces.To the best of our knowledge,this paper is a first attempt to study the ASPC criterion on continuous-time MDPs with Polish state and action spaces.The corresponding transition rates are allowed to be unbounded,and the cost rates may have neither upper nor lower bounds.Under some mild hypotheses,we prove the existence of e (ε ≥ 0)-ASPC optimal stationary policies based on two different approaches:one is the “optimality equation” approach and the other is the “two optimality inequalities” approach.

  1. "Markov at the bat": a model of cognitive processing in baseball batters.

    Science.gov (United States)

    Gray, Rob

    2002-11-01

    Anecdotal evidence from players and coaches indicates that cognitive processing (e.g., expectations about the upcoming pitch) plays an important role in successful baseball batting, yet this aspect of hitting has not been investigated in detail. The present study provides experimental evidence that prior expectations significantly influence the timing of a baseball swing. A two-state Markov model was used to predict the effects of pitch sequence and pitch count on batting performance. The model is a hitting strategy of switching between expectancy states using a simple set of transition rules. In a simulated batting experiment, the model provided a good fit to the hitting performance of 6 experienced college baseball players, and the estimated model parameters were highly correlated with playing level.

  2. Approximate Policy Iteration with a Policy Language Bias: Solving Relational Markov Decision Processes

    CERN Document Server

    Fern, A; Yoon, S; 10.1613/jair.1700

    2011-01-01

    We study an approach to policy selection for large relational Markov Decision Processes (MDPs). We consider a variant of approximate policy iteration (API) that replaces the usual value-function learning step with a learning step in policy space. This is advantageous in domains where good policies are easier to represent and learn than the corresponding value functions, which is often the case for the relational MDPs we are interested in. In order to apply API to such problems, we introduce a relational policy language and corresponding learner. In addition, we introduce a new bootstrapping routine for goal-based planning domains, based on random walks. Such bootstrapping is necessary for many large relational MDPs, where reward is extremely sparse, as API is ineffective in such domains when initialized with an uninformed policy. Our experiments show that the resulting system is able to find good policies for a number of classical planning domains and their stochastic variants by solving them as extremely lar...

  3. Averaging for a Fully-Coupled Piecewise Deterministic Markov Process in Infinite Dimension

    CERN Document Server

    Genadot, Alexandre

    2011-01-01

    In this paper, we consider the generalized Hodgkin-Huxley model introduced by Austin in \\cite{Austin}. This model describes the propagation of an action potential along the axon of a neuron at the scale of ion channels. Mathematically, this model is a fully-coupled Piecewise Deterministic Markov Process (PDMP) in infinite dimension. We introduce two time scales in this model in considering that some ion channels open and close at faster jump rates than others. We perform a slow-fast analysis of this model and prove that asymptotically this two time scales model reduces to the so called averaged model which is still a PDMP in infinite dimension for which we provide effective evolution equations and jump rates.

  4. Dynamic Request Routing for Online Video-on-Demand Service: A Markov Decision Process Approach

    Directory of Open Access Journals (Sweden)

    Jianxiong Wan

    2014-01-01

    Full Text Available We investigate the request routing problem in the CDN-based Video-on-Demand system. We model the system as a controlled queueing system including a dispatcher and several edge servers. The system is formulated as a Markov decision process (MDP. Since the MDP formulation suffers from the so-called “the curse of dimensionality” problem, we then develop a greedy heuristic algorithm, which is simple and can be implemented online, to approximately solve the MDP model. However, we do not know how far it deviates from the optimal solution. To address this problem, we further aggregate the state space of the original MDP model and use the bounded-parameter MDP (BMDP to reformulate the system. This allows us to obtain a suboptimal solution with a known performance bound. The effectiveness of two approaches is evaluated in a simulation study.

  5. A Gallavotti-Cohen-Evans-Morriss Like Symmetry for a Class of Markov Jump Processes

    Science.gov (United States)

    Barato, Andre Cardoso; Chetrite, Raphaël; Hinrichsen, Haye; Mukamel, David

    2012-01-01

    We investigate a new symmetry of the large deviation function of certain time-integrated currents in non-equilibrium systems. The symmetry is similar to the well-known Gallavotti-Cohen-Evans-Morriss-symmetry for the entropy production, but it concerns a different functional of the stochastic trajectory. The symmetry can be found in a restricted class of Markov jump processes, where the network of microscopic transitions has a particular structure and the transition rates satisfy certain constraints. We provide three physical examples, where time-integrated observables display such a symmetry. Moreover, we argue that the origin of the symmetry can be traced back to time-reversal if stochastic trajectories are grouped appropriately.

  6. Recent advances in nonlinear speech processing

    CERN Document Server

    Faundez-Zanuy, Marcos; Esposito, Antonietta; Cordasco, Gennaro; Drugman, Thomas; Solé-Casals, Jordi; Morabito, Francesco

    2016-01-01

    This book presents recent advances in nonlinear speech processing beyond nonlinear techniques. It shows that it exploits heuristic and psychological models of human interaction in order to succeed in the implementations of socially believable VUIs and applications for human health and psychological support. The book takes into account the multifunctional role of speech and what is “outside of the box” (see Björn Schuller’s foreword). To this aim, the book is organized in 6 sections, each collecting a small number of short chapters reporting advances “inside” and “outside” themes related to nonlinear speech research. The themes emphasize theoretical and practical issues for modelling socially believable speech interfaces, ranging from efforts to capture the nature of sound changes in linguistic contexts and the timing nature of speech; labors to identify and detect speech features that help in the diagnosis of psychological and neuronal disease, attempts to improve the effectiveness and performa...

  7. Quantum Information Processing using Nonlinear Optical Effects

    DEFF Research Database (Denmark)

    Andersen, Lasse Mejling

    of the converted idler depends on the other pump. This allows for temporal-mode-multiplexing. When the effects of nonlinear phase modulation (NPM) are included, the phases of the natural input and output modes are changed, reducing the separability. These effects are to some degree mediated by pre......This PhD thesis treats applications of nonlinear optical effects for quantum information processing. The two main applications are four-wave mixing in the form of Bragg scattering (BS) for quantum-state-preserving frequency conversion, and sum-frequency generation (SFG) in second-order nonlinear...... to obtain a 100 % conversion efficiency is to use multiple stages of frequency conversion, but this setup suffers from the combined effects of NPM. This problem is circumvented by using asymmetrically pumped BS, where one pump is continuous wave. For this setup, NPM is found to only lead to linear phase...

  8. Incorporating risk attitude into Markov-process decision models: importance for individual decision making.

    Science.gov (United States)

    Cher, D J; Miyamoto, J; Lenert, L A

    1997-01-01

    Most decision models published in the medical literature take a risk-neutral perspective. Under risk neutrality, the utility of a gamble is equivalent to its expected value and the marginal utility of living a given unit of time is the same regardless of when it occurs. Most patients, however, are not risk-neutral. Not only does risk aversion affect decision analyses when tradeoffs between short- and long-term survival are involved, it also affects the interpretation of time-tradeoff measures of health-state utility. The proportional time tradeoff under- or overestimates the disutility of an inferior health state, depending on whether the patient is risk-seeking or risk-averse (it is unbiased if the patient is risk-neutral). The authors review how risk attitude with respect to gambles for survival duration can be incorporated into decision models using the framework of risk-adjusted quality-adjusted life years (RA-QALYs). They present a simple extension of this framework that allows RA-QALYs to be calculated for Markov-process decision models. Using a previously published Markov-process model of surgical vs expectant treatment for benign prostatic hypertrophy (BPH), they show how attitude towards risk affects the expected number of QALYs calculated by the model. In this model, under risk neutrality, surgery was the preferred option. Under mild risk aversion, expectant treatment was the preferred option. Risk attitude is an important aspect of preferences that should be incorporated into decision models where one treatment option has upfront risks of morbidity or mortality.

  9. Wind Farm Reliability Modelling Using Bayesian Networks and Semi-Markov Processes

    Directory of Open Access Journals (Sweden)

    Robert Adam Sobolewski

    2015-09-01

    Full Text Available Technical reliability plays an important role among factors affecting the power output of a wind farm. The reliability is determined by an internal collection grid topology and reliability of its electrical components, e.g. generators, transformers, cables, switch breakers, protective relays, and busbars. A wind farm reliability’s quantitative measure can be the probability distribution of combinations of operating and failed states of the farm’s wind turbines. The operating state of a wind turbine is its ability to generate power and to transfer it to an external power grid, which means the availability of the wind turbine and other equipment necessary for the power transfer to the external grid. This measure can be used for quantitative analysis of the impact of various wind farm topologies and the reliability of individual farm components on the farm reliability, and for determining the expected farm output power with consideration of the reliability. This knowledge may be useful in an analysis of power generation reliability in power systems. The paper presents probabilistic models that quantify the wind farm reliability taking into account the above-mentioned technical factors. To formulate the reliability models Bayesian networks and semi-Markov processes were used. Using Bayesian networks the wind farm structural reliability was mapped, as well as quantitative characteristics describing equipment reliability. To determine the characteristics semi-Markov processes were used. The paper presents an example calculation of: (i probability distribution of the combination of both operating and failed states of four wind turbines included in the wind farm, and (ii expected wind farm output power with consideration of its reliability.

  10. Markov Model Applied to Gene Evolution

    Institute of Scientific and Technical Information of China (English)

    季星来; 孙之荣

    2001-01-01

    The study of nucleotide substitution is very important both to our understanding of gene evolution and to reliable estimation of phylogenetic relationships. In this paper nucleotide substitution is assumed to be random and the Markov model is applied to the study of the evolution of genes. Then a non-linear optimization approach is proposed for estimating substitution in real sequences. This substitution is called the "Nucleotide State Transfer Matrix". One of the most important conclusions from this work is that gene sequence evolution conforms to the Markov process. Also, some theoretical evidences for random evolution are given from energy analysis of DNA replication.

  11. Research on Multi-Stage Inventory Model by Markov Decision Process

    Science.gov (United States)

    Rong, Ke

    This paper researched multi-stage inventory system and established limited inventory Markov model, on the other hand it induced DP algorithm of limited inventory Markov model. The results proved that the reorder point of multi-stage inventory system can guarantee demand, and also allows the storage costs to a minimum level in accordance with the above model.

  12. Effects of stochastic interest rates in decision making under risk: A Markov decision process model for forest management

    Science.gov (United States)

    Mo Zhou; Joseph Buongiorno

    2011-01-01

    Most economic studies of forest decision making under risk assume a fixed interest rate. This paper investigated some implications of this stochastic nature of interest rates. Markov decision process (MDP) models, used previously to integrate stochastic stand growth and prices, can be extended to include variable interest rates as well. This method was applied to...

  13. Forward recursion for Markov decision processes with skip-free-to-the-right transitions - Part II : Non-standard applications

    NARCIS (Netherlands)

    Wijngaard, J.; Stidham, S.

    2000-01-01

    This paper is the sequel to Wijngaard and Stidham (1986). The topic is a countable state, average reward semi-Markov decision process with a transition mechanism that is skip-free to the right. The applications are controlled GI/M/1 queues. Skip-free to the right means that state n cannot be reached

  14. Flexible Sampling of Discrete Scale Invariant Markov Processes: Covariance and Spectrum

    CERN Document Server

    Modarresi, N

    2010-01-01

    In this paper we consider some flexible discrete sampling of a discrete scale invariant process $\\{X(t), t\\in{\\bf R^+}\\}$ with scale $l>1$. By this method we plan to have $q$ samples at arbitrary points ${\\bf s}_0, {\\bf s}_1,..., {\\bf s}_{q-1}$ in interval $[1, l)$ and proceed our sampling in the intervals $[l^n, l^{n+1})$ at points $l^n{\\bf s}_0, l^n{\\bf s}_1,..., l^n{\\bf s}_{q-1}$, $n\\in {\\bf Z}$. Thus we have a discrete time scale invariant (DT-SI) process and introduce an embedded DT-SI process as $W(nq+k)=X(l^n{\\bf s}_k)$, $q\\in {\\bf N}$, $k= 0,..., q-1$. We also consider $V(n)=\\big(V^0(n),..., V^{q-1}(n)\\big)$ where $V^k(n)=W(nq+k)$, as an embedded $q$-dimensional discrete time self-similar (DT-SS) process. By introducing quasi Lamperti transformation, we find spectral representation of such process and its spectral density matrix is given. Finally by imposing wide sense Markov property for $W(\\cdot)$ and $V(\\cdot)$, we show that the spectral density matrix of $V(\\cdot)$ and spectral density function of...

  15. On Markov Chains and Filtrations

    OpenAIRE

    1997-01-01

    In this paper we rederive some well known results for continuous time Markov processes that live on a finite state space.Martingale techniques are used throughout the paper. Special attention is paid to the construction of a continuous timeMarkov process, when we start from a discrete time Markov chain. The Markov property here holds with respect tofiltrations that need not be minimal.

  16. An information theoretic approach for generating an aircraft avoidance Markov Decision Process

    Science.gov (United States)

    Weinert, Andrew J.

    Developing a collision avoidance system that can meet safety standards required of commercial aviation is challenging. A dynamic programming approach to collision avoidance has been developed to optimize and generate logics that are robust to the complex dynamics of the national airspace. The current approach represents the aircraft avoidance problem as Markov Decision Processes and independently optimizes a horizontal and vertical maneuver avoidance logics. This is a result of the current memory requirements for each logic, simply combining the logics will result in a significantly larger representation. The "curse of dimensionality" makes it computationally inefficient and unfeasible to optimize this larger representation. However, existing and future collision avoidance systems have mostly defined the decision process by hand. In response, a simulation-based framework was built to better understand how each potential state quantifies the aircraft avoidance problem with regards to safety and operational components. The framework leverages recent advances in signals processing and database, while enabling the highest fidelity analysis of Monte Carlo aircraft encounter simulations to date. This framework enabled the calculation of how well each state of the decision process quantifies the collision risk and the associated memory requirements. Using this analysis, a collision avoidance logic that leverages both horizontal and vertical actions was built and optimized using this simulation based approach.

  17. Internal Decoupling in Nonlinear Process Control

    Directory of Open Access Journals (Sweden)

    Jens G. Balchen

    1988-07-01

    Full Text Available A simple method has been investigated for the total or partial removal of the effect of non-linear process phenomena in multi-variable feedback control systems. The method is based upon computing the control variables which will drive the process at desired rates. It is shown that the effect of model errors in the linearization of the process can be partly removed through the use of large feedback gains. In practice there will be limits on how large gains can he used. The sensitivity to parameter errors is less pronounced and the transient behaviour is superior to that of ordinary PI controllers.

  18. Using Bayesian Nonparametric Hidden Semi-Markov Models to Disentangle Affect Processes during Marital Interaction.

    Directory of Open Access Journals (Sweden)

    William A Griffin

    Full Text Available Sequential affect dynamics generated during the interaction of intimate dyads, such as married couples, are associated with a cascade of effects-some good and some bad-on each partner, close family members, and other social contacts. Although the effects are well documented, the probabilistic structures associated with micro-social processes connected to the varied outcomes remain enigmatic. Using extant data we developed a method of classifying and subsequently generating couple dynamics using a Hierarchical Dirichlet Process Hidden semi-Markov Model (HDP-HSMM. Our findings indicate that several key aspects of existing models of marital interaction are inadequate: affect state emissions and their durations, along with the expected variability differences between distressed and nondistressed couples are present but highly nuanced; and most surprisingly, heterogeneity among highly satisfied couples necessitate that they be divided into subgroups. We review how this unsupervised learning technique generates plausible dyadic sequences that are sensitive to relationship quality and provide a natural mechanism for computational models of behavioral and affective micro-social processes.

  19. Evaluation of linearly solvable Markov decision process with dynamic model learning in a mobile robot navigation task.

    Science.gov (United States)

    Kinjo, Ken; Uchibe, Eiji; Doya, Kenji

    2013-01-01

    Linearly solvable Markov Decision Process (LMDP) is a class of optimal control problem in which the Bellman's equation can be converted into a linear equation by an exponential transformation of the state value function (Todorov, 2009b). In an LMDP, the optimal value function and the corresponding control policy are obtained by solving an eigenvalue problem in a discrete state space or an eigenfunction problem in a continuous state using the knowledge of the system dynamics and the action, state, and terminal cost functions. In this study, we evaluate the effectiveness of the LMDP framework in real robot control, in which the dynamics of the body and the environment have to be learned from experience. We first perform a simulation study of a pole swing-up task to evaluate the effect of the accuracy of the learned dynamics model on the derived the action policy. The result shows that a crude linear approximation of the non-linear dynamics can still allow solution of the task, despite with a higher total cost. We then perform real robot experiments of a battery-catching task using our Spring Dog mobile robot platform. The state is given by the position and the size of a battery in its camera view and two neck joint angles. The action is the velocities of two wheels, while the neck joints were controlled by a visual servo controller. We test linear and bilinear dynamic models in tasks with quadratic and Guassian state cost functions. In the quadratic cost task, the LMDP controller derived from a learned linear dynamics model performed equivalently with the optimal linear quadratic regulator (LQR). In the non-quadratic task, the LMDP controller with a linear dynamics model showed the best performance. The results demonstrate the usefulness of the LMDP framework in real robot control even when simple linear models are used for dynamics learning.

  20. A novel thermodynamic state recursion method for description of nonideal nonlinear chromatographic process of frontal analysis.

    Science.gov (United States)

    Liu, Qian; OuYang, Liangfei; Liang, Heng; Li, Nan; Geng, Xindu

    2012-06-01

    A novel thermodynamic state recursion (TSR) method, which is based on nonequilibrium thermodynamic path described by the Lagrangian-Eulerian representation, is presented to simulate the whole chromatographic process of frontal analysis using the spatial distribution of solute bands in time series like as a series of images. TSR differs from the current numerical methods using the partial differential equations in Eulerian representation. The novel method is used to simulate the nonideal, nonlinear hydrophobic interaction chromatography (HIC) processes of lysozyme and myoglobin under the discrete complex boundary conditions. The results show that the simulated breakthrough curves agree well with the experimental ones. The apparent diffusion coefficient and the Langmuir isotherm parameters of the two proteins in HIC are obtained by the state recursion inverse method. Due to its the time domain and Markov characteristics, TSR is applicable to the design and online control of the nonlinear multicolumn chromatographic systems.

  1. Markovian Shock Models, Deterioration Processes, Stratified Markov Processes and Replacement Policies.

    Science.gov (United States)

    1986-05-05

    ideas concerning the nature of high Reynolds number, fully developed turbulence are presented and the possible roles of singular solutions and "fuzzy...filamentation in cannot be fathomed by legalese nonlinear optics are also candidates. For example, so we apply Occarn’s razor singular solutions of the...Reynolds num- An appealing idea of modern dynamics is that hers. Singular solutions like defects and disloca- the complicated and apparently

  2. Processing Approach of Non-linear Adjustment Models in the Space of Non-linear Models

    Institute of Scientific and Technical Information of China (English)

    LI Chaokui; ZHU Qing; SONG Chengfang

    2003-01-01

    This paper investigates the mathematic features of non-linear models and discusses the processing way of non-linear factors which contributes to the non-linearity of a nonlinear model. On the basis of the error definition, this paper puts forward a new adjustment criterion, SGPE.Last, this paper investigates the solution of a non-linear regression model in the non-linear model space and makes the comparison between the estimated values in non-linear model space and those in linear model space.

  3. Clarification of basic factorization identity is for the almost semi-continuous latticed Poisson processes on the Markov chain

    Directory of Open Access Journals (Sweden)

    Gerich M. S.

    2012-12-01

    Full Text Available Let ${xi(t, x(t}$ be a homogeneous semi-continuous lattice Poisson process on the Markov chain.The jumps of one sign are geometrically distributed, and jumps of the opposite sign are arbitrary latticed distribution. For a suchprocesses the relations for the components of two-sided matrix factorization are established.This relations define the moment genereting functions for extremumf of the process and their complements.

  4. Clarification of basic factorization identity is for the almost semi-continuous latticed Poisson processes on the Markov chain

    OpenAIRE

    2012-01-01

    Let ${xi(t), x(t)}$ be a homogeneous semi-continuous lattice Poisson process on the Markov chain.The jumps of one sign are geometrically distributed, and jumps of the opposite sign are arbitrary latticed distribution. For a suchprocesses the relations for the components of two-sided matrix factorization are established.This relations define the moment genereting functions for extremumf of the process and their complements.

  5. Nonlinear Process Fault Diagnosis Based on Serial Principal Component Analysis.

    Science.gov (United States)

    Deng, Xiaogang; Tian, Xuemin; Chen, Sheng; Harris, Chris J

    2016-12-22

    Many industrial processes contain both linear and nonlinear parts, and kernel principal component analysis (KPCA), widely used in nonlinear process monitoring, may not offer the most effective means for dealing with these nonlinear processes. This paper proposes a new hybrid linear-nonlinear statistical modeling approach for nonlinear process monitoring by closely integrating linear principal component analysis (PCA) and nonlinear KPCA using a serial model structure, which we refer to as serial PCA (SPCA). Specifically, PCA is first applied to extract PCs as linear features, and to decompose the data into the PC subspace and residual subspace (RS). Then, KPCA is performed in the RS to extract the nonlinear PCs as nonlinear features. Two monitoring statistics are constructed for fault detection, based on both the linear and nonlinear features extracted by the proposed SPCA. To effectively perform fault identification after a fault is detected, an SPCA similarity factor method is built for fault recognition, which fuses both the linear and nonlinear features. Unlike PCA and KPCA, the proposed method takes into account both linear and nonlinear PCs simultaneously, and therefore, it can better exploit the underlying process's structure to enhance fault diagnosis performance. Two case studies involving a simulated nonlinear process and the benchmark Tennessee Eastman process demonstrate that the proposed SPCA approach is more effective than the existing state-of-the-art approach based on KPCA alone, in terms of nonlinear process fault detection and identification.

  6. Safety-cost trade-offs in medical device reuse: a Markov decision process model.

    Science.gov (United States)

    Sloan, Thomas W

    2007-02-01

    Healthcare expenditures in the US are approaching 2 trillion dollars, and hospitals and other healthcare providers are under tremendous pressure to rein in costs. One cost-saving approach which is gaining popularity is the reuse of medical devices which were designed only for a single use. Device makers decry this practice as unsanitary and unsafe, but a growing number of third-party firms are willing to sterilize, refurbish, and/or remanufacture devices and resell them to hospitals at a fraction of the original price. Is this practice safe? Is reliance on single-use devices sustainable? A Markov decision process (MDP) model is formulated to study the trade-offs involved in these decisions. Several key parameters are examined: device costs, device failure probabilities, and failure penalty cost. For each of these parameters, expressions are developed which identify the indifference point between using new and reprocessed devices. The results can be used to inform the debate on the economic, ethical, legal, and environmental dimensions of this complex issue.

  7. A Markov decision process approach to temporal modulation of dose fractions in radiation therapy planning.

    Science.gov (United States)

    Kim, M; Ghate, A; Phillips, M H

    2009-07-21

    The current state of the art in cancer treatment by radiation optimizes beam intensity spatially such that tumors receive high dose radiation whereas damage to nearby healthy tissues is minimized. It is common practice to deliver the radiation over several weeks, where the daily dose is a small constant fraction of the total planned. Such a 'fractionation schedule' is based on traditional models of radiobiological response where normal tissue cells possess the ability to repair sublethal damage done by radiation. This capability is significantly less prominent in tumors. Recent advances in quantitative functional imaging and biological markers are providing new opportunities to measure patient response to radiation over the treatment course. This opens the door for designing fractionation schedules that take into account the patient's cumulative response to radiation up to a particular treatment day in determining the fraction on that day. We propose a novel approach that, for the first time, mathematically explores the benefits of such fractionation schemes. This is achieved by building a stylistic Markov decision process (MDP) model, which incorporates some key features of the problem through intuitive choices of state and action spaces, as well as transition probability and reward functions. The structure of optimal policies for this MDP model is explored through several simple numerical examples.

  8. Analyses of Markov decision process structure regarding the possible strategic use of interacting memory systems.

    Science.gov (United States)

    Zilli, Eric A; Hasselmo, Michael E

    2008-01-01

    Behavioral tasks are often used to study the different memory systems present in humans and animals. Such tasks are usually designed to isolate and measure some aspect of a single memory system. However, it is not necessarily clear that any given task actually does isolate a system or that the strategy used by a subject in the experiment is the one desired by the experimenter. We have previously shown that when tasks are written mathematically as a form of partially observable Markov decision processes, the structure of the tasks provide information regarding the possible utility of certain memory systems. These previous analyses dealt with the disambiguation problem: given a specific ambiguous observation of the environment, is there information provided by a given memory strategy that can disambiguate that observation to allow a correct decision? Here we extend this approach to cases where multiple memory systems can be strategically combined in different ways. Specifically, we analyze the disambiguation arising from three ways by which episodic-like memory retrieval might be cued (by another episodic-like memory, by a semantic association, or by working memory for some earlier observation). We also consider the disambiguation arising from holding earlier working memories, episodic-like memories or semantic associations in working memory. From these analyses we can begin to develop a quantitative hierarchy among memory systems in which stimulus-response memories and semantic associations provide no disambiguation while the episodic memory system provides the most flexible disambiguation, with working memory at an intermediate level.

  9. Optimizing Prescription of Chinese Herbal Medicine for Unstable Angina Based on Partially Observable Markov Decision Process

    Directory of Open Access Journals (Sweden)

    Yan Feng

    2013-01-01

    Full Text Available Objective. Initial optimized prescription of Chinese herb medicine for unstable angina (UA. Methods. Based on partially observable Markov decision process model (POMDP, we choose hospitalized patients of 3 syndrome elements, such as qi deficiency, blood stasis, and turbid phlegm for the data mining, analysis, and objective evaluation of the diagnosis and treatment of UA at a deep level in order to optimize the prescription of Chinese herb medicine for UA. Results. The recommended treatment options of UA for qi deficiency, blood stasis, and phlegm syndrome patients were as follows: Milkvetch Root + Tangshen + Indian Bread + Largehead Atractylodes Rhizome (ADR=0.96630; Danshen Root + Chinese Angelica + Safflower + Red Peony Root + Szechwan Lovage Rhizome Orange Fruit (ADR=0.76; Snakegourd Fruit + Longstamen Onion Bulb + Pinellia Tuber + Dried Tangerine peel + Largehead Atractylodes Rhizome + Platycodon Root (ADR=0.658568. Conclusion. This study initially optimized prescriptions for UA based on POMDP, which can be used as a reference for further development of UA prescription in Chinese herb medicine.

  10. Strategic level proton therapy patient admission planning: a Markov decision process modeling approach.

    Science.gov (United States)

    Gedik, Ridvan; Zhang, Shengfan; Rainwater, Chase

    2016-01-25

    A relatively new consideration in proton therapy planning is the requirement that the mix of patients treated from different categories satisfy desired mix percentages. Deviations from these percentages and their impacts on operational capabilities are of particular interest to healthcare planners. In this study, we investigate intelligent ways of admitting patients to a proton therapy facility that maximize the total expected number of treatment sessions (fractions) delivered to patients in a planning period with stochastic patient arrivals and penalize the deviation from the patient mix restrictions. We propose a Markov Decision Process (MDP) model that provides very useful insights in determining the best patient admission policies in the case of an unexpected opening in the facility (i.e., no-shows, appointment cancellations, etc.). In order to overcome the curse of dimensionality for larger and more realistic instances, we propose an aggregate MDP model that is able to approximate optimal patient admission policies using the worded weight aggregation technique. Our models are applicable to healthcare treatment facilities throughout the United States, but are motivated by collaboration with the University of Florida Proton Therapy Institute (UFPTI).

  11. Markov-CA model using analytical hierarchy process and multiregression technique

    Science.gov (United States)

    Omar, N. Q.; Sanusi, S. A. M.; Hussin, W. M. W.; Samat, N.; Mohammed, K. S.

    2014-06-01

    The unprecedented increase in population and rapid rate of urbanisation has led to extensive land use changes. Cellular automata (CA) are increasingly used to simulate a variety of urban dynamics. This paper introduces a new CA based on an integration model built-in multi regression and multi-criteria evaluation to improve the representation of CA transition rule. This multi-criteria evaluation is implemented by utilising data relating to the environmental and socioeconomic factors in the study area in order to produce suitability maps (SMs) using an analytical hierarchical process, which is a well-known method. Before being integrated to generate suitability maps for the periods from 1984 to 2010 based on the different decision makings, which have become conditioned for the next step of CA generation. The suitability maps are compared in order to find the best maps based on the values of the root equation (R2). This comparison can help the stakeholders make better decisions. Thus, the resultant suitability map derives a predefined transition rule for the last step for CA model. The approach used in this study highlights a mechanism for monitoring and evaluating land-use and land-cover changes in Kirkuk city, Iraq owing changes in the structures of governments, wars, and an economic blockade over the past decades. The present study asserts the high applicability and flexibility of Markov-CA model. The results have shown that the model and its interrelated concepts are performing rather well.

  12. Value-directed human behavior analysis from video using partially observable Markov decision processes.

    Science.gov (United States)

    Hoey, Jesse; Little, James J

    2007-07-01

    This paper presents a method for learning decision theoretic models of human behaviors from video data. Our system learns relationships between the movements of a person, the context in which they are acting, and a utility function. This learning makes explicit that the meaning of a behavior to an observer is contained in its relationship to actions and outcomes. An agent wishing to capitalize on these relationships must learn to distinguish the behaviors according to how they help the agent to maximize utility. The model we use is a partially observable Markov decision process, or POMDP. The video observations are integrated into the POMDP using a dynamic Bayesian network that creates spatial and temporal abstractions amenable to decision making at the high level. The parameters of the model are learned from training data using an a posteriori constrained optimization technique based on the expectation-maximization algorithm. The system automatically discovers classes of behaviors and determines which are important for choosing actions that optimize over the utility of possible outcomes. This type of learning obviates the need for labeled data from expert knowledge about which behaviors are significant and removes bias about what behaviors may be useful to recognize in a particular situation. We show results in three interactions: a single player imitation game, a gestural robotic control problem, and a card game played by two people.

  13. Optimizing prescription of chinese herbal medicine for unstable angina based on partially observable markov decision process.

    Science.gov (United States)

    Feng, Yan; Qiu, Yu; Zhou, Xuezhong; Wang, Yixin; Xu, Hao; Liu, Baoyan

    2013-01-01

    Objective. Initial optimized prescription of Chinese herb medicine for unstable angina (UA). Methods. Based on partially observable Markov decision process model (POMDP), we choose hospitalized patients of 3 syndrome elements, such as qi deficiency, blood stasis, and turbid phlegm for the data mining, analysis, and objective evaluation of the diagnosis and treatment of UA at a deep level in order to optimize the prescription of Chinese herb medicine for UA. Results. The recommended treatment options of UA for qi deficiency, blood stasis, and phlegm syndrome patients were as follows: Milkvetch Root + Tangshen + Indian Bread + Largehead Atractylodes Rhizome (ADR = 0.96630); Danshen Root + Chinese Angelica + Safflower + Red Peony Root + Szechwan Lovage Rhizome Orange Fruit (ADR = 0.76); Snakegourd Fruit + Longstamen Onion Bulb + Pinellia Tuber + Dried Tangerine peel + Largehead Atractylodes Rhizome + Platycodon Root (ADR = 0.658568). Conclusion. This study initially optimized prescriptions for UA based on POMDP, which can be used as a reference for further development of UA prescription in Chinese herb medicine.

  14. Composition of web services using Markov decision processes and dynamic programming.

    Science.gov (United States)

    Uc-Cetina, Víctor; Moo-Mena, Francisco; Hernandez-Ucan, Rafael

    2015-01-01

    We propose a Markov decision process model for solving the Web service composition (WSC) problem. Iterative policy evaluation, value iteration, and policy iteration algorithms are used to experimentally validate our approach, with artificial and real data. The experimental results show the reliability of the model and the methods employed, with policy iteration being the best one in terms of the minimum number of iterations needed to estimate an optimal policy, with the highest Quality of Service attributes. Our experimental work shows how the solution of a WSC problem involving a set of 100,000 individual Web services and where a valid composition requiring the selection of 1,000 services from the available set can be computed in the worst case in less than 200 seconds, using an Intel Core i5 computer with 6 GB RAM. Moreover, a real WSC problem involving only 7 individual Web services requires less than 0.08 seconds, using the same computational power. Finally, a comparison with two popular reinforcement learning algorithms, sarsa and Q-learning, shows that these algorithms require one or two orders of magnitude and more time than policy iteration, iterative policy evaluation, and value iteration to handle WSC problems of the same complexity.

  15. Real-time network security situation visualization and threat assessment based on semi-Markov process

    Science.gov (United States)

    Chen, Junhua

    2013-03-01

    To cope with a large amount of data in current sensed environments, decision aid tools should provide their understanding of situations in a time-efficient manner, so there is an increasing need for real-time network security situation awareness and threat assessment. In this study, the state transition model of vulnerability in the network based on semi-Markov process is proposed at first. Once events are triggered by an attacker's action or system response, the current states of the vulnerabilities are known. Then we calculate the transition probabilities of the vulnerability from the current state to security failure state. Furthermore in order to improve accuracy of our algorithms, we adjust the probabilities that they exploit the vulnerability according to the attacker's skill level. In the light of the preconditions and post-conditions of vulnerabilities in the network, attack graph is built to visualize security situation in real time. Subsequently, we predict attack path, recognize attack intention and estimate the impact through analysis of attack graph. These help administrators to insight into intrusion steps, determine security state and assess threat. Finally testing in a network shows that this method is reasonable and feasible, and can undertake tremendous analysis task to facilitate administrators' work.

  16. Analysis of the non-Markov parameter in continuous-time signal processing.

    Science.gov (United States)

    Varghese, J J; Bellette, P A; Weegink, K J; Bradley, A P; Meehan, P A

    2014-02-01

    The use of statistical complexity metrics has yielded a number of successful methodologies to differentiate and identify signals from complex systems where the underlying dynamics cannot be calculated. The Mori-Zwanzig framework from statistical mechanics forms the basis for the generalized non-Markov parameter (NMP). The NMP has been used to successfully analyze signals in a diverse set of complex systems. In this paper we show that the Mori-Zwanzig framework masks an elegantly simple closed form of the first NMP, which, for C(1) smooth autocorrelation functions, is solely a function of the second moment (spread) and amplitude envelope of the measured power spectrum. We then show that the higher-order NMPs can be constructed in closed form in a modular fashion from the lower-order NMPs. These results provide an alternative, signal processing-based perspective to analyze the NMP, which does not require an understanding of the Mori-Zwanzig generating equations. We analyze the parametric sensitivity of the zero-frequency value of the first NMP, which has been used as a metric to discriminate between states in complex systems. Specifically, we develop closed-form expressions for three instructive systems: band-limited white noise, the output of white noise input to an idealized all-pole filter,f and a simple harmonic oscillator driven by white noise. Analysis of these systems shows a primary sensitivity to the decay rate of the tail of the power spectrum.

  17. Polynomial Time Algorithms for Branching Markov Decision Processes and Probabilistic Min(Max) Polynomial Bellman Equations

    CERN Document Server

    Etessami, Kousha; Yannakakis, Mihalis

    2012-01-01

    We show that one can approximate the least fixed point solution for a multivariate system of monotone probabilistic max(min) polynomial equations, referred to as maxPPSs (and minPPSs, respectively), in time polynomial in both the encoding size of the system of equations and in log(1/epsilon), where epsilon > 0 is the desired additive error bound of the solution. (The model of computation is the standard Turing machine model.) We establish this result using a generalization of Newton's method which applies to maxPPSs and minPPSs, even though the underlying functions are only piecewise-differentiable. This generalizes our recent work which provided a P-time algorithm for purely probabilistic PPSs. These equations form the Bellman optimality equations for several important classes of infinite-state Markov Decision Processes (MDPs). Thus, as a corollary, we obtain the first polynomial time algorithms for computing to within arbitrary desired precision the optimal value vector for several classes of infinite-state...

  18. A markov decision process model for the optimal dispatch of military medical evacuation assets.

    Science.gov (United States)

    Keneally, Sean K; Robbins, Matthew J; Lunday, Brian J

    2016-06-01

    We develop a Markov decision process (MDP) model to examine aerial military medical evacuation (MEDEVAC) dispatch policies in a combat environment. The problem of deciding which aeromedical asset to dispatch to each service request is complicated by the threat conditions at the service locations and the priority class of each casualty event. We assume requests for MEDEVAC support arrive sequentially, with the location and the priority of each casualty known upon initiation of the request. The United States military uses a 9-line MEDEVAC request system to classify casualties as being one of three priority levels: urgent, priority, and routine. Multiple casualties can be present at a single casualty event, with the highest priority casualty determining the priority level for the casualty event. Moreover, an armed escort may be required depending on the threat level indicated by the 9-line MEDEVAC request. The proposed MDP model indicates how to optimally dispatch MEDEVAC helicopters to casualty events in order to maximize steady-state system utility. The utility gained from servicing a specific request depends on the number of casualties, the priority class for each of the casualties, and the locations of both the servicing ambulatory helicopter and casualty event. Instances of the dispatching problem are solved using a relative value iteration dynamic programming algorithm. Computational examples are used to investigate optimal dispatch policies under different threat situations and armed escort delays; the examples are based on combat scenarios in which United States Army MEDEVAC units support ground operations in Afghanistan.

  19. A Markov Decision Process Model for Cervical Cancer Screening Policies in Colombia.

    Science.gov (United States)

    Akhavan-Tabatabaei, Raha; Sánchez, Diana Marcela; Yeung, Thomas G

    2017-02-01

    Cervical cancer is the second most common cancer in women around the world, and the human papillomavirus (HPV) is universally known as the necessary agent for developing this disease. Through early detection of abnormal cells and HPV virus types, cervical cancer incidents can be reduced and disease progression prevented. We propose a finite-horizon Markov decision process model to determine the optimal screening policies for cervical cancer prevention. The optimal decision is given in terms of when and what type of screening test to be performed on a patient based on her current diagnosis, age, HPV contraction risk, and screening test results. The cost function considers the tradeoff between the cost of prevention and treatment procedures and the risk of taking no action while taking into account a cost assigned to loss of life quality in each state. We apply the model to data collected from a representative sample of 1141 affiliates at a health care provider located in Bogotá, Colombia. To track the disease incidence more effectively and avoid higher cancer rates and future costs, the optimal policies recommend more frequent colposcopies and Pap tests for women with riskier profiles.

  20. Markov decision processes and the belief-desire-intention model bridging the gap for autonomous agents

    CERN Document Server

    Simari, Gerardo I

    2011-01-01

    In this work, we provide a treatment of the relationship between two models that have been widely used in the implementation of autonomous agents: the Belief DesireIntention (BDI) model and Markov Decision Processes (MDPs). We start with an informal description of the relationship, identifying the common features of the two approaches and the differences between them. Then we hone our understanding of these differences through an empirical analysis of the performance of both models on the TileWorld testbed. This allows us to show that even though the MDP model displays consistently better behavior than the BDI model for small worlds, this is not the case when the world becomes large and the MDP model cannot be solved exactly. Finally we present a theoretical analysis of the relationship between the two approaches, identifying mappings that allow us to extract a set of intentions from a policy (a solution to an MDP), and to extract a policy from a set of intentions.

  1. Analyses of Markov decision process structure regarding the possible strategic use of interacting memory systems

    Directory of Open Access Journals (Sweden)

    Eric A Zilli

    2008-12-01

    Full Text Available Behavioral tasks are often used to study the different memory systems present in humans and animals. Such tasks are usually designed to isolate and measure some aspect of a single memory system. However, it is not necessarily clear that any given task actually does isolate a system or that the strategy used by a subject in the experiment is the one desired by the experimenter. We have previously shown that when tasks are written mathematically as a form of partially-observable Markov decision processes, the structure of the tasks provide information regarding the possible utility of certain memory systems. These previous analyses dealt with the disambiguation problem: given a specific ambiguous observation of the environment, is there information provided by a given memory strategy that can disambiguate that observation to allow a correct decisionµ Here we extend this approach to cases where multiple memory systems can be strategically combined in different ways. Specifically, we analyze the disambiguation arising from three ways by which episodic-like memory retrieval might be cued (by another episodic-like memory, by a semantic association, or by working memory for some earlier observation. We also consider the disambiguation arising from holding earlier working memories, episodic-like memories or semantic associations in working memory. From these analyses we can begin to develop a quantitative hierarchy among memory systems in which stimulus-response memories and semantic associations provide no disambiguation while the episodic memory system provides the most flexible

  2. Dynamics of a tracer granular particle as a nonequilibrium Markov process

    Science.gov (United States)

    Puglisi, Andrea; Visco, Paolo; Trizac, Emmanuel; van Wijland, Frédéric

    2006-02-01

    The dynamics of a tracer particle in a stationary driven granular gas is investigated. We show how to transform the linear Boltzmann equation, describing the dynamics of the tracer into a master equation for a continuous Markov process. The transition rates depend on the stationary velocity distribution of the gas. When the gas has a Gaussian velocity probability distribution function (PDF), the stationary velocity PDF of the tracer is Gaussian with a lower temperature and satisfies detailed balance for any value of the restitution coefficient α . As soon as the velocity PDF of the gas departs from the Gaussian form, detailed balance is violated. This nonequilibrium state can be characterized in terms of a Lebowitz-Spohn action functional W(τ) defined over trajectories of time duration τ . We discuss the properties of this functional and of a similar functional Wmacr (τ) , which differs from the first for a term that is nonextensive in time. On the one hand, we show that in numerical experiments (i.e., at finite times τ ), the two functionals have different fluctuations and Wmacr always satisfies an Evans-Searles-like symmetry. On the other hand, we cannot observe the verification of the Lebowitz-Spohn-Gallavotti-Cohen (LS-GC) relation, which is expected for W(τ) at very large times τ . We give an argument for the possible failure of the LS-GC relation in this situation. We also suggest practical recipes for measuring W(τ) and Wmacr (τ) in experiments.

  3. An open Markov chain scheme model for a credit consumption portfolio fed by ARIMA and SARMA processes

    Science.gov (United States)

    Esquível, Manuel L.; Fernandes, José Moniz; Guerreiro, Gracinda R.

    2016-06-01

    We introduce a schematic formalism for the time evolution of a random population entering some set of classes and such that each member of the population evolves among these classes according to a scheme based on a Markov chain model. We consider that the flow of incoming members is modeled by a time series and we detail the time series structure of the elements in each of the classes. We present a practical application to data from a credit portfolio of a Cape Verdian bank; after modeling the entering population in two different ways - namely as an ARIMA process and as a deterministic sigmoid type trend plus a SARMA process for the residues - we simulate the behavior of the population and compare the results. We get that the second method is more accurate in describing the behavior of the populations when compared to the observed values in a direct simulation of the Markov chain.

  4. Alternative decision modelling techniques for the evaluation of health care technologies: Markov processes versus discrete event simulation.

    Science.gov (United States)

    Karnon, Jonathan

    2003-10-01

    Markov models have traditionally been used to evaluate the cost-effectiveness of competing health care technologies that require the description of patient pathways over extended time horizons. Discrete event simulation (DES) is a more flexible, but more complicated decision modelling technique, that can also be used to model extended time horizons. Through the application of a Markov process and a DES model to an economic evaluation comparing alternative adjuvant therapies for early breast cancer, this paper compares the respective processes and outputs of these alternative modelling techniques. DES displays increased flexibility in two broad areas, though the outputs from the two modelling techniques were similar. These results indicate that the use of DES may be beneficial only when the available data demonstrates particular characteristics.

  5. A sow replacement model using Bayesian updating in a three-level hierarchic Markov process. I. Biological model

    DEFF Research Database (Denmark)

    Kristensen, Anders Ringgaard; Søllested, Thomas Algot

    2004-01-01

    that really uses all these methodological improvements. In this paper, the biological model describing the performance and feed intake of sows is presented. In particular, estimation of herd specific parameters is emphasized. The optimization model is described in a subsequent paper......Several replacement models have been presented in literature. In other applicational areas like dairy cow replacement, various methodological improvements like hierarchical Markov processes and Bayesian updating have been implemented, but not in sow models. Furthermore, there are methodological...... improvements like multi-level hierarchical Markov processes with decisions on multiple time scales, efficient methods for parameter estimations at herd level and standard software that has been hardly implemented at all in any replacement model. The aim of this study is to present a sow replacement model...

  6. Projected metastable Markov processes and their estimation with observable operator models

    Energy Technology Data Exchange (ETDEWEB)

    Wu, Hao, E-mail: hao.wu@fu-berlin.de; Prinz, Jan-Hendrik, E-mail: jan-hendrik.prinz@fu-berlin.de; Noé, Frank, E-mail: frank.noe@fu-berlin.de [DFG Research Center Matheon, Free University Berlin, Arnimallee 6, 14195 Berlin (Germany)

    2015-10-14

    The determination of kinetics of high-dimensional dynamical systems, such as macromolecules, polymers, or spin systems, is a difficult and generally unsolved problem — both in simulation, where the optimal reaction coordinate(s) are generally unknown and are difficult to compute, and in experimental measurements, where only specific coordinates are observable. Markov models, or Markov state models, are widely used but suffer from the fact that the dynamics on a coarsely discretized state spaced are no longer Markovian, even if the dynamics in the full phase space are. The recently proposed projected Markov models (PMMs) are a formulation that provides a description of the kinetics on a low-dimensional projection without making the Markovianity assumption. However, as yet no general way of estimating PMMs from data has been available. Here, we show that the observed dynamics of a PMM can be exactly described by an observable operator model (OOM) and derive a PMM estimator based on the OOM learning.

  7. Collaborative distributed sensor management and information exchange flow control for multitarget tracking using Markov decision processes

    Science.gov (United States)

    Akselrod, Dimitry; Kirubarajan, T.

    2008-04-01

    In this paper, we consider the problem of collaborative management of uninhabited aerial vehicles (UAVs) for multitarget tracking. In addition to providing a solution to the problem of controlling individual UAVs, we present a method for controlling the information flow among them. The latter provides a solution to one of the main problems in decentralized tracking, namely, distributed information transfer and fusion among the participating platforms. The problem of decentralized cooperative control considered in this paper is an optimization of the information obtained by a number of UAVs, carrying out surveillance over a region, which includes a number of confirmed and suspected moving targets with the goal to track confirmed targets and detects new targets in the area. Each UAV has to decide on the most optimal path with the objective to track as many targets as possible, maximizing the information obtained during its operation with the maximum possible accuracy at the lowest possible cost. Limited communication between UAVs and uncertainty in the information obtained by each UAV regarding the location of the ground targets are addressed in the problem formulation. In order to handle these issues, the problem is presented as an operation of a group of decision makers. Markov Decision Processes (MDPs) are incorporated into the solution. A decision mechanism for collaborative distributed data fusion provides each UAV with the required data for the fusion process while substantially reducing redundancy in the information flow in the overall system. We consider a distributed data fusion system consisting of UAVs that are decentralized, heterogenous, and potentially unreliable. Simulation results are presented on a representative multisensor-multitarget tracking problem.

  8. A distance for probability spaces, and long-term values in Markov Decision Processes and Repeated Games

    CERN Document Server

    Renault, Jérôme

    2012-01-01

    Given a finite set $K$, we denote by $X=\\Delta(K)$ the set of probabilities on $K$ and by $Z=\\Delta_f(X)$ the set of Borel probabilities on $X$ with finite support. Studying a Markov Decision Process with partial information on $K$ naturally leads to a Markov Decision Process with full information on $X$. We introduce a new metric $d_*$ on $Z$ such that the transitions become 1-Lipschitz from $(X, \\|.\\|_1)$ to $(Z,d_*)$. In the first part of the article, we define and prove several properties of the metric $d_*$. Especially, $d_*$ satisfies a Kantorovich-Rubinstein type duality formula and can be characterized by using disintegrations. In the second part, we characterize the limit values in several classes of "compact non expansive" Markov Decision Processes. In particular we use the metric $d_*$ to characterize the limit value in Partial Observation MDP with finitely many states and in Repeated Games with an informed controller with finite sets of states and actions. Moreover in each case we can prove the ex...

  9. Algorithmic analysis of the maximum level length in general-block two-dimensional Markov processes

    Directory of Open Access Journals (Sweden)

    2006-01-01

    Full Text Available Two-dimensional continuous-time Markov chains (CTMCs are useful tools for studying stochastic models such as queueing, inventory, and production systems. Of particular interest in this paper is the distribution of the maximal level visited in a busy period because this descriptor provides an excellent measure of the system congestion. We present an algorithmic analysis for the computation of its distribution which is valid for Markov chains with general-block structure. For a multiserver batch arrival queue with retrials and negative arrivals, we exploit the underlying internal block structure and present numerical examples that reveal some interesting facts of the system.

  10. Nonlinear Stochastic Dynamics of Complex Systems, II: Potential of Entropic Force in Markov Systems with Nonequilibrium Steady State, Generalized Gibbs Function and Criticality

    CERN Document Server

    Thompson, Lowell F

    2016-01-01

    In this paper we revisit the notion of the "minus logarithm of stationary probability" as a generalized potential in nonequilibrium systems and attempt to illustrate its central role in an axiomatic approach to stochastic nonequilibrium thermodynamics of complex systems. It is demonstrated that this quantity arises naturally through both monotonicity results of Markov processes and as the rate function when a stochastic process approaches a detrministic limit. We then undertake a more detailed mathematical analysis of the consequences of this quantity, culminating in a necessary and sufficient condition for the criticality of stochastic systems. This condition is then discussed in the context of recent results about criticality in biological systems.

  11. Comparative effectiveness research on patients with acute ischemic stroke using Markov decision processes

    Directory of Open Access Journals (Sweden)

    Wu Darong

    2012-03-01

    Full Text Available Abstract Background Several methodological issues with non-randomized comparative clinical studies have been raised, one of which is whether the methods used can adequately identify uncertainties that evolve dynamically with time in real-world systems. The objective of this study is to compare the effectiveness of different combinations of Traditional Chinese Medicine (TCM treatments and combinations of TCM and Western medicine interventions in patients with acute ischemic stroke (AIS by using Markov decision process (MDP theory. MDP theory appears to be a promising new method for use in comparative effectiveness research. Methods The electronic health records (EHR of patients with AIS hospitalized at the 2nd Affiliated Hospital of Guangzhou University of Chinese Medicine between May 2005 and July 2008 were collected. Each record was portioned into two "state-action-reward" stages divided by three time points: the first, third, and last day of hospital stay. We used the well-developed optimality technique in MDP theory with the finite horizon criterion to make the dynamic comparison of different treatment combinations. Results A total of 1504 records with a primary diagnosis of AIS were identified. Only states with more than 10 (including 10 patients' information were included, which gave 960 records to be enrolled in the MDP model. Optimal combinations were obtained for 30 types of patient condition. Conclusion MDP theory makes it possible to dynamically compare the effectiveness of different combinations of treatments. However, the optimal interventions obtained by the MDP theory here require further validation in clinical practice. Further exploratory studies with MDP theory in other areas in which complex interventions are common would be worthwhile.

  12. Learning to maximize reward rate: a model based on semi-Markov decision processes.

    Science.gov (United States)

    Khodadadi, Arash; Fakhari, Pegah; Busemeyer, Jerome R

    2014-01-01

    WHEN ANIMALS HAVE TO MAKE A NUMBER OF DECISIONS DURING A LIMITED TIME INTERVAL, THEY FACE A FUNDAMENTAL PROBLEM: how much time they should spend on each decision in order to achieve the maximum possible total outcome. Deliberating more on one decision usually leads to more outcome but less time will remain for other decisions. In the framework of sequential sampling models, the question is how animals learn to set their decision threshold such that the total expected outcome achieved during a limited time is maximized. The aim of this paper is to provide a theoretical framework for answering this question. To this end, we consider an experimental design in which each trial can come from one of the several possible "conditions." A condition specifies the difficulty of the trial, the reward, the penalty and so on. We show that to maximize the expected reward during a limited time, the subject should set a separate value of decision threshold for each condition. We propose a model of learning the optimal value of decision thresholds based on the theory of semi-Markov decision processes (SMDP). In our model, the experimental environment is modeled as an SMDP with each "condition" being a "state" and the value of decision thresholds being the "actions" taken in those states. The problem of finding the optimal decision thresholds then is cast as the stochastic optimal control problem of taking actions in each state in the corresponding SMDP such that the average reward rate is maximized. Our model utilizes a biologically plausible learning algorithm to solve this problem. The simulation results show that at the beginning of learning the model choses high values of decision threshold which lead to sub-optimal performance. With experience, however, the model learns to lower the value of decision thresholds till finally it finds the optimal values.

  13. Collaborative distributed sensor management for multitarget tracking using hierarchical Markov decision processes

    Science.gov (United States)

    Akselrod, D.; Sinha, A.; Kirubarajan, T.

    2007-09-01

    In this paper, we consider the problem of collaborative sensor management with particular application to using unmanned aerial vehicles (UAVs) for multitarget tracking. The problem of decentralized cooperative control considered in this paper is an optimization of the information obtained by a number of unmanned aerial vehicles (UAVs) equipped with Ground Moving Target Indicator (GMTI) radars, carrying out surveillance over a region which includes a number of confirmed and suspected moving targets. The goal is to track confirmed targets and detect new targets in the area. Each UAV has to decide on the most optimal path with the objective to track as many targets as possible maximizing the information obtained during its operation with the maximum possible accuracy at the lowest possible cost. Limited communication between UAVs and uncertainty in the information obtained by each UAV regarding the location of the ground targets are addressed in the problem formulation. In order to handle these issues, the problem is presented as a decentralized operation of a group of decision-makers lacking full observability of the global state of the system. Markov Decision Processes (MDPs) are incorporated into the solution. Given the MDP model, a local policy of actions for a single agent (UAV) is given by a mapping from a current partial view of a global state observed by an agent to actions. The available probability model regarding possible and confirmed locations of the targets is considered in the computations of the UAVs' policies. The authors present multi-level hierarchy of MDPs controlling each of the UAVs. Each level in the hierarchy solves a problem at a different level of abstraction. Simulation results are presented on a representative multisensor-multitarget tracking problem.

  14. Nonlinear biochemical signal processing via noise propagation.

    Science.gov (United States)

    Kim, Kyung Hyuk; Qian, Hong; Sauro, Herbert M

    2013-10-14

    Single-cell studies often show significant phenotypic variability due to the stochastic nature of intra-cellular biochemical reactions. When the numbers of molecules, e.g., transcription factors and regulatory enzymes, are in low abundance, fluctuations in biochemical activities become significant and such "noise" can propagate through regulatory cascades in terms of biochemical reaction networks. Here we develop an intuitive, yet fully quantitative method for analyzing how noise affects cellular phenotypes based on identifying a system's nonlinearities and noise propagations. We observe that such noise can simultaneously enhance sensitivities in one behavioral region while reducing sensitivities in another. Employing this novel phenomenon we designed three biochemical signal processing modules: (a) A gene regulatory network that acts as a concentration detector with both enhanced amplitude and sensitivity. (b) A non-cooperative positive feedback system, with a graded dose-response in the deterministic case, that serves as a bistable switch due to noise-induced ultra-sensitivity. (c) A noise-induced linear amplifier for gene regulation that requires no feedback. The methods developed in the present work allow one to understand and engineer nonlinear biochemical signal processors based on fluctuation-induced phenotypes.

  15. Applying a Markov approach as a Lean Thinking analysis of waste elimination in a Rice Production Process

    Directory of Open Access Journals (Sweden)

    Eldon Glen Caldwell Marin

    2015-01-01

    Full Text Available The Markov Chains Model was proposed to analyze stochastic events when recursive cycles occur; for example, when rework in a continuous flow production affects the overall performance. Typically, the analysis of rework and scrap is done through a wasted material cost perspective and not from the perspective of waste capacity that reduces throughput and economic value added (EVA. Also, we can not find many cases of this application in agro-industrial production in Latin America, given the complexity of the calculations and the need for robust applications. This scientific work presents the results of a quasi-experimental research approach in order to explain how to apply DOE methods and Markov analysis in a rice production process located in Central America, evaluating the global effects of a single reduction in rework and scrap in a part of the whole line. The results show that in this case it is possible to evaluate benefits from Global Throughput and EVA perspective and not only from the saving costs perspective, finding a relationship between operational indicators and corporate performance. However, it was found that it is necessary to analyze the markov chains configuration with many rework points, also it is still relevant to take into account the effects on takt time and not only scrap´s costs.

  16. Global satisfactory control for nonlinear integrator processes with long delay

    Institute of Scientific and Technical Information of China (English)

    Yiqun YANG; Guobo XIANG

    2007-01-01

    Integrator processes with long delay are difficult to control. Nonlinear characteristics of actuators make the control problem more challenging. A technique is proposed in this paper for global satisfactory control (GSC) of such processes with relay-type nonlinearity. An oscillatory control signal is injected into the nonlinear process; the amplitude and frequency of the oscillatory signal are designed to linearise the nonlinear process in the sense of harmonic analysis; and a state feedback controller is configured to implement GSC over the linearised process. An illustrative example is given to demonstrate the effectiveness of the proposed method.

  17. Framework based on Markov modulated Poisson processes for modeling traffic with long-range dependence

    Science.gov (United States)

    Ferreira Salvador, Paulo J.; Valadas, Rui J. M. T.

    2001-07-01

    This paper proposes a novel fitting procedure for Markov Modulated Poisson Processes (MMPPs), consisting of the superposition of N 2-MMPPs, that is capable of capturing the long-range characteristics of the traffic. The procedure matches both the autocovariance and marginal distribution functions of the rate process. We start by matching each 2-MMPP to a different component of the autocovariance function. We then map the parameters of the model with N individual 2-MMPPs (termed superposed MMPP) to the parameters of the equivalent MMPP with 2N states that results from the superposition of the N individual 2-MMPPs (termed generic MMPP). Finally, the parameters of the generic MMPP are fitted to the marginal distribution, subject to the constraints imposed by the autocovariance matching. Specifically, the matching of the distribution will be restricted by the fact that it may not be possible to decompose a generic MMPP back into individual 2-MMPPs. Overall, our procedure is motivated by the fact that direct relationships can be established between the autocovariance and the parameters of the superposed MMPP and between the marginal distribution and the parameters of the generic MMPP. We apply the fitting procedure to traffic traces exhibiting LRD including (i) IP traffic measured at our institution and (ii) IP traffic traces available in the Internet such as the well known, publicly available, Bellcore traces. The selected traces are representative of a wide range of services/protocols used in the Internet. We assess the fitting procedure by comparing the measured and fitted traces (traces generated from the fitted models) in terms of (i) Hurst parameter; (ii) degree of approximation between the autocovariance and marginal distribution curves; (iii) range of time scales where LRD is observed using a wavelet based estimator and (iv) packet loss ratio suffered in a single buffer for different values of the buffer capacity. Results are very clear in showing that MMPPs

  18. Nonlinear calibration transfer based on hierarchical Bayesian models and Lagrange Multipliers: Error bounds of estimates via Monte Carlo - Markov Chain sampling.

    Science.gov (United States)

    Seichter, Felicia; Vogt, Josef; Radermacher, Peter; Mizaikoff, Boris

    2017-01-25

    The calibration of analytical systems is time-consuming and the effort for daily calibration routines should therefore be minimized, while maintaining the analytical accuracy and precision. The 'calibration transfer' approach proposes to combine calibration data already recorded with actual calibrations measurements. However, this strategy was developed for the multivariate, linear analysis of spectroscopic data, and thus, cannot be applied to sensors with a single response channel and/or a non-linear relationship between signal and desired analytical concentration. To fill this gap for a non-linear calibration equation, we assume that the coefficients for the equation, collected over several calibration runs, are normally distributed. Considering that coefficients of an actual calibration are a sample of this distribution, only a few standards are needed for a complete calibration data set. The resulting calibration transfer approach is demonstrated for a fluorescence oxygen sensor and implemented as a hierarchical Bayesian model, combined with a Lagrange Multipliers technique and Monte-Carlo Markov-Chain sampling. The latter provides realistic estimates for coefficients and prediction together with accurate error bounds by simulating known measurement errors and system fluctuations. Performance criteria for validation and optimal selection of a reduced set of calibration samples were developed and lead to a setup which maintains the analytical performance of a full calibration. Strategies for a rapid determination of problems occurring in a daily calibration routine, are proposed, thereby opening the possibility of correcting the problem just in time.

  19. Spectral methods for bivariate Markov processes with diffusion and discrete components and a variant of the Wright-Fisher model

    CERN Document Server

    de la Iglesia, Manuel D

    2011-01-01

    The aim of this paper is to study differential and spectral properties of the infinitesimal operator of two dimensional Markov processes with diffusion and discrete components. The infinitesimal operator is now a second-order differential operator with matrix-valued coefficients, from which we can derive backward and forward equations, a spectral representation of the probability density, study recurrence of the process and the corresponding invariant distribution. All these results are applied to an example coming from group representation theory which can be viewed as a variant of the Wright-Fisher model involving only mutation effects.

  20. Markov switching in GARCH processes and mean reverting stock market volatility

    OpenAIRE

    Dueker, Michael J.

    1995-01-01

    This paper introduces four models of conditional heteroskedasticity that contain markov switching parameters to examine their multi-period stock-market volatility forecasts as predictions of options-implied volatilities. The volatility model that best predicts the behavior of the optionsimplied volatilities allows the student-t degrees-of-freedom parameter to switch such that the conditional variance and kurtosis are subject to discrete shifts. The half-life of the most leptokurtic state is e...

  1. Artificial intelligence framework for simulating clinical decision-making: a Markov decision process approach.

    Science.gov (United States)

    Bennett, Casey C; Hauser, Kris

    2013-01-01

    In the modern healthcare system, rapidly expanding costs/complexity, the growing myriad of treatment options, and exploding information streams that often do not effectively reach the front lines hinder the ability to choose optimal treatment decisions over time. The goal in this paper is to develop a general purpose (non-disease-specific) computational/artificial intelligence (AI) framework to address these challenges. This framework serves two potential functions: (1) a simulation environment for exploring various healthcare policies, payment methodologies, etc., and (2) the basis for clinical artificial intelligence - an AI that can "think like a doctor". This approach combines Markov decision processes and dynamic decision networks to learn from clinical data and develop complex plans via simulation of alternative sequential decision paths while capturing the sometimes conflicting, sometimes synergistic interactions of various components in the healthcare system. It can operate in partially observable environments (in the case of missing observations or data) by maintaining belief states about patient health status and functions as an online agent that plans and re-plans as actions are performed and new observations are obtained. This framework was evaluated using real patient data from an electronic health record. The results demonstrate the feasibility of this approach; such an AI framework easily outperforms the current treatment-as-usual (TAU) case-rate/fee-for-service models of healthcare. The cost per unit of outcome change (CPUC) was $189 vs. $497 for AI vs. TAU (where lower is considered optimal) - while at the same time the AI approach could obtain a 30-35% increase in patient outcomes. Tweaking certain AI model parameters could further enhance this advantage, obtaining approximately 50% more improvement (outcome change) for roughly half the costs. Given careful design and problem formulation, an AI simulation framework can approximate optimal

  2. A Markov decision process approach to multi-category patient scheduling in a diagnostic facility.

    Science.gov (United States)

    Gocgun, Yasin; Bresnahan, Brian W; Ghate, Archis; Gunn, Martin L

    2011-10-01

    To develop a mathematical model for multi-category patient scheduling decisions in computed tomography (CT), and to investigate associated tradeoffs from economic and operational perspectives. We modeled this decision-problem as a finite-horizon Markov decision process (MDP) with expected net CT revenue as the performance metric. The performance of optimal policies was compared with five heuristics using data from an urban hospital. In addition to net revenue, other patient-throughput and service-quality metrics were also used in this comparative analysis. The optimal policy had a threshold structure in the two-scanner case - it prioritized one type of patient when the queue-length for that type exceeded a threshold. The net revenue gap between the optimal policy and the heuristics ranged from 5% to 12%. This gap was 4% higher in the more congested, single-scanner system than in the two-scanner system. The performance of the net revenue maximizing policy was similar to the heuristics, when compared with respect to the alternative performance metrics in the two-scanner case. Under the optimal policy, the average number of patients that were not scanned by the end of the day, and the average patient waiting-time, were both nearly 80% smaller in the two-scanner case than in the single-scanner case. The net revenue gap between the optimal policy and the priority-based heuristics was nearly 2% smaller as compared to the first-come-first-served and random selection schemes. Net revenue was most sensitive to inpatient (IP) penalty costs in the single-scanner system, whereas to IP and outpatient revenues in the two-scanner case. The performance of the optimal policy is competitive with the operational and economic metrics considered in this paper. Such a policy can be implemented relatively easily and could be tested in practice in the future. The priority-based heuristics are next-best to the optimal policy and are much easier to implement. Copyright © 2011 Elsevier B

  3. [Classification of human sleep stages based on EEG processing using hidden Markov models].

    Science.gov (United States)

    Doroshenkov, L G; Konyshev, V A; Selishchev, S V

    2007-01-01

    The goal of this work was to describe an automated system for classification of human sleep stages. Classification of sleep stages is an important problem of diagnosis and treatment of human sleep disorders. The developed classification method is based on calculation of characteristics of the main sleep rhythms. It uses hidden Markov models. The method is highly accurate and provides reliable identification of the main stages of sleep. The results of automatic classification are in good agreement with the results of sleep stage identification performed by an expert somnologist using Rechtschaffen and Kales rules. This substantiates the applicability of the developed classification system to clinical diagnosis.

  4. Signal processing of MEMS gyroscope arrays to improve accuracy using a 1st order Markov for rate signal modeling.

    Science.gov (United States)

    Jiang, Chengyu; Xue, Liang; Chang, Honglong; Yuan, Guangmin; Yuan, Weizheng

    2012-01-01

    This paper presents a signal processing technique to improve angular rate accuracy of the gyroscope by combining the outputs of an array of MEMS gyroscope. A mathematical model for the accuracy improvement was described and a Kalman filter (KF) was designed to obtain optimal rate estimates. Especially, the rate signal was modeled by a first-order Markov process instead of a random walk to improve overall performance. The accuracy of the combined rate signal and affecting factors were analyzed using a steady-state covariance. A system comprising a six-gyroscope array was developed to test the presented KF. Experimental tests proved that the presented model was effective at improving the gyroscope accuracy. The experimental results indicated that six identical gyroscopes with an ARW noise of 6.2 °/√h and a bias drift of 54.14 °/h could be combined into a rate signal with an ARW noise of 1.8 °/√h and a bias drift of 16.3 °/h, while the estimated rate signal by the random walk model has an ARW noise of 2.4 °/√h and a bias drift of 20.6 °/h. It revealed that both models could improve the angular rate accuracy and have a similar performance in static condition. In dynamic condition, the test results showed that the first-order Markov process model could reduce the dynamic errors 20% more than the random walk model.

  5. Control policies for a water-treatment system using the Markov Decision Process.

    Science.gov (United States)

    Chiam, Tze; Mitchell, Cary; Yih, Yuehwern

    the system's current state but not the "path" that it has taken. Due to this "memoryless" property and the stochastic properties of the system, the state transition can be modeled by the Markov process. A reward system was constructed to assign reward values to every state visited. A water system is considered to be in a "good" state when it has sufficient clean water to meet the demands of crewmembers. Such states will receive a much higher reward value than states in which crewmembers suffer from water deficiencies. Transition probabilities are obtained through simulation using the Markovian model. Nine policies based on different values of treatment efficiencies for both subsystems were defined. One policy is applied to the system at every hour. The choice of policy to apply affects the system behavior (and state). Hence, it is important to apply a policy that is "best" for the system every hour. The Policy Iteration algorithm is used for this purpose. This algorithm provides the best policy under steady-state conditions. The transition probabilities and reward values are formulated into appropriate mathematical representation and are solved by applying the Policy Iteration algorithm. A system that uses the best policy is compared against one that uses a fixed policy by the use of a paired-t test. Results show that a system applying best policies has statistically better performance than a system operating on a fixed policy. This methodology is also applicable to various other scenarios with different system design, magnitude of "stochastic-ness", including system modules such as the crop system. Research sponsored in part by NASA grant NAG5-12686.

  6. Stochastic Model Checking Continuous Time Markov Process%随机模型检测连续时间Markov过程

    Institute of Scientific and Technical Information of China (English)

    钮俊; 曾国荪; 吕新荣; 徐畅

    2011-01-01

    The trustworthiness of a dynamic system includes the correctness of function and the satisfiability of per formance mainly. This paper proposed an approach to verify the function and performance of a system under considera tion integratedly. Continuous-time Markov decision process (CTMDP) is a model that contains some aspects such as probabilistic choice;stochastic timing and nondeterminacy; and it is the model by which we verify function properties and analyze performance properties uniformly. We can verify the functional and performance specifications by computing the reachability probabilities in the product CTMDP. We proved the correctness of our approach; and obtained our veri fication results by using model checker MRMC(Markov Reward Model Checker). The theoretical results show that model checking CTMDP model is necessary and the model checking approach is feasible.%功能正确和性能可满足是复杂系统可信要求非常重要的两个方面.从定性验证和定量分析相结合的角度,对复杂并发系统进行功能验证和性能分析,统一地评估系统是否可信.连续时间Markov决策过程CTMDP(Continuous-time Markov decision process)能够统一刻画复杂系统的概率选择、随机时间及不确定性等重要特征.提出用CTMDP作为系统定性验证和定量分析模型,将复杂系统的功能验证和性能分析转化为CTMDP中的可达概率求解,并证明验证过程的正确性,最终借助模型检测器MRMC(Markov Reward Model Checker)实现模型检测.理论分析表明,提出的针对CTMDP模型的验证需求是必要的,验证思路和方法具有可行性.

  7. Numerical simulations of piecewise deterministic Markov processes with an application to the stochastic Hodgkin-Huxley model

    Science.gov (United States)

    Ding, Shaojie; Qian, Min; Qian, Hong; Zhang, Xuejuan

    2016-12-01

    The stochastic Hodgkin-Huxley model is one of the best-known examples of piecewise deterministic Markov processes (PDMPs), in which the electrical potential across a cell membrane, V(t), is coupled with a mesoscopic Markov jump process representing the stochastic opening and closing of ion channels embedded in the membrane. The rates of the channel kinetics, in turn, are voltage-dependent. Due to this interdependence, an accurate and efficient sampling of the time evolution of the hybrid stochastic systems has been challenging. The current exact simulation methods require solving a voltage-dependent hitting time problem for multiple path-dependent intensity functions with random thresholds. This paper proposes a simulation algorithm that approximates an alternative representation of the exact solution by fitting the log-survival function of the inter-jump dwell time, H(t), with a piecewise linear one. The latter uses interpolation points that are chosen according to the time evolution of the H(t), as the numerical solution to the coupled ordinary differential equations of V(t) and H(t). This computational method can be applied to all PDMPs. Pathwise convergence of the approximated sample trajectories to the exact solution is proven, and error estimates are provided. Comparison with a previous algorithm that is based on piecewise constant approximation is also presented.

  8. Numerical simulations of piecewise deterministic Markov processes with an application to the stochastic Hodgkin-Huxley model.

    Science.gov (United States)

    Ding, Shaojie; Qian, Min; Qian, Hong; Zhang, Xuejuan

    2016-12-28

    The stochastic Hodgkin-Huxley model is one of the best-known examples of piecewise deterministic Markov processes (PDMPs), in which the electrical potential across a cell membrane, V(t), is coupled with a mesoscopic Markov jump process representing the stochastic opening and closing of ion channels embedded in the membrane. The rates of the channel kinetics, in turn, are voltage-dependent. Due to this interdependence, an accurate and efficient sampling of the time evolution of the hybrid stochastic systems has been challenging. The current exact simulation methods require solving a voltage-dependent hitting time problem for multiple path-dependent intensity functions with random thresholds. This paper proposes a simulation algorithm that approximates an alternative representation of the exact solution by fitting the log-survival function of the inter-jump dwell time, H(t), with a piecewise linear one. The latter uses interpolation points that are chosen according to the time evolution of the H(t), as the numerical solution to the coupled ordinary differential equations of V(t) and H(t). This computational method can be applied to all PDMPs. Pathwise convergence of the approximated sample trajectories to the exact solution is proven, and error estimates are provided. Comparison with a previous algorithm that is based on piecewise constant approximation is also presented.

  9. 基于 Markov 链互模拟的航天器发射任务可靠度模型%Mission reliability model of spacecraft launch based on bisimulation of continuous-time Markov processes

    Institute of Scientific and Technical Information of China (English)

    董学军; 武小悦; 陈英武

    2012-01-01

    状态空间复杂、多过程并发执行和子过程反复迭代的特点,使航天器发射工程实施全过程的任务可靠性评估难以量化.通过构建多个并发执行的时间连续的 Markov 链对航天器发射工程状态转移约束关系进行描述,采用互模拟时间等价关系简化航天器发射工程实施过程的状态空间,利用连续时间 Markov 链的概率转移特性进行建模与分析,得到了全系统、全过程的航天器发射任务可靠度模型.数值验证表明该模型可用于航天器发射任务工期推演、可靠度评估以及薄弱环节分析.%Characteristics of complex state space, multi-process concurrent execution and sub-processes iterative make mission reliability assessment for the whole process of spacecraft launch engineering implementation is difficult to quantify. Multiple concurrently executing continuous time Markov chain is constructed to describe state transition constraint relations of spacecraft launch engineering. The state space of the whole process of spacecraft launch engineering implementation is simplified by bisimulation equivalence relation. The model of mission reliability for spacecraft launch engineering is builded by continuous time Markov chain transfer probability characteristics. In this paper, the example applied results shows that the model is a feasible for decision-making demonstration of spacecraft launch project, evaluation of mission reliability and analysis of weak link.

  10. DOUBLE-MARKOV RISK MODEL

    Institute of Scientific and Technical Information of China (English)

    Xiaoyun MO; Jieming ZHOU; Hui OU; Xiangqun YANG

    2013-01-01

    Given a new Double-Markov risk model DM =(μ,Q,v,H; Y,Z) and Double-Markov risk process U ={U(t),t ≥ 0}.The ruin or survival problem is addressed.Equations which the survival probability satisfied and the formulas of calculating survival probability are obtained.Recursion formulas of calculating the survival probability and analytic expression of recursion items are obtained.The conclusions are expressed by Q matrix for a Markov chain and transition probabilities for another Markov Chain.

  11. Nonlinear spectral unmixing of hyperspectral images using Gaussian processes

    CERN Document Server

    Altmann, Yoann; McLaughlin, Steve; Tourneret, Jean-Yves

    2012-01-01

    This paper presents an unsupervised algorithm for nonlinear unmixing of hyperspectral images. The proposed model assumes that the pixel reflectances result from a nonlinear function of the abundance vectors associated with the pure spectral components. We assume that the spectral signatures of the pure components and the nonlinear function are unknown. The first step of the proposed method consists of the Bayesian estimation of the abundance vectors for all the image pixels and the nonlinear function relating the abundance vectors to the observations. The endmembers are subsequently estimated using Gaussian process regression. The performance of the unmixing strategy is evaluated with simulations conducted on synthetic and real data.

  12. On a Result for Finite Markov Chains

    Science.gov (United States)

    Kulathinal, Sangita; Ghosh, Lagnojita

    2006-01-01

    In an undergraduate course on stochastic processes, Markov chains are discussed in great detail. Textbooks on stochastic processes provide interesting properties of finite Markov chains. This note discusses one such property regarding the number of steps in which a state is reachable or accessible from another state in a finite Markov chain with M…

  13. Dynamic access clustering selecting mechanism based on Markov decision process for MANET

    Institute of Scientific and Technical Information of China (English)

    WANG Dao-yuan; TIAN Hui

    2007-01-01

    Clustering is an important method in the mobile Ad-hoc network (MANET). As a result of their mobility, the cluster selection is inevitable for the mobile nodes during their roaming between the different clusters. In this study, based on the analysis of the cluster-selecting problem in the environment containing multiple clusters, which are overlaying and intercrossing, a novel dynamic selecting mechanism is proposed to resolve the dynamic selection optimization of roaming between the different clusters in MANET. This selecting mechanism is also based on the consideration of the stability of communication system, the communicating bandwidth, and the effect of cluster selecting on the communication and also in accordance with the Markov decision-making model.

  14. MARKOV PROCESSES IN MODELING LAND USE AND LAND COVER CHANGES IN SINTRA-CASCAIS, PORTUGAL

    Directory of Open Access Journals (Sweden)

    PEDRO CABRAL

    2009-01-01

    Full Text Available En este artículo los procesos de alteración de la utilización y ocupación del suelo (LUCC son investigados recorriendo-se a técnicas de teledetección y a cadenas de Markov en lasmunicipalidades de Sintra y Cascais (Portugal entre los anos de 1989 y 2000. El papel del Parque Natural de Sintra-Cascais (PNSC es evaluado.Los resultados demuestran que, dentro del PNSC, el LUCC presente depende del pasadoinmediato del uso y ocupación del suelo siguiendo un comportamiento Markoviano. Fuera del PNSC, LUCC es aleatorio y no sigue un proceso Markoviano. Estimativas del LUCC para el ano de 2006 son presentadas para el área dentro del PNSC. Estos resultados refuerzan el papel del PNSC como una herramienta indispensable para preservar la estabilidad del LUCC y garantizar sus funciones.

  15. Bubble nonlinear dynamics and stimulated scattering process

    Science.gov (United States)

    Jie, Shi; De-Sen, Yang; Sheng-Guo, Shi; Bo, Hu; Hao-Yang, Zhang; Shi-Yong, Hu

    2016-02-01

    A complete understanding of the bubble dynamics is deemed necessary in order to achieve their full potential applications in industry and medicine. For this purpose it is first needed to expand our knowledge of a single bubble behavior under different possible conditions including the frequency and pressure variations of the sound field. In addition, stimulated scattering of sound on a bubble is a special effect in sound field, and its characteristics are associated with bubble oscillation mode. A bubble in liquid can be considered as a representative example of nonlinear dynamical system theory with its resonance, and its dynamics characteristics can be described by the Keller-Miksis equation. The nonlinear dynamics of an acoustically excited gas bubble in water is investigated by using theoretical and numerical analysis methods. Our results show its strongly nonlinear behavior with respect to the pressure amplitude and excitation frequency as the control parameters, and give an intuitive insight into stimulated sound scattering on a bubble. It is seen that the stimulated sound scattering is different from common dynamical behaviors, such as bifurcation and chaos, which is the result of the nonlinear resonance of a bubble under the excitation of a high amplitude acoustic sound wave essentially. The numerical analysis results show that the threshold of stimulated sound scattering is smaller than those of bifurcation and chaos in the common condition. Project supported by the Program for Changjiang Scholars and Innovative Research Team in University, China (Grant No. IRT1228) and the Young Scientists Fund of the National Natural Science Foundation of China (Grant Nos. 11204050 and 11204049).

  16. Phasic Triplet Markov Chains.

    Science.gov (United States)

    El Yazid Boudaren, Mohamed; Monfrini, Emmanuel; Pieczynski, Wojciech; Aïssani, Amar

    2014-11-01

    Hidden Markov chains have been shown to be inadequate for data modeling under some complex conditions. In this work, we address the problem of statistical modeling of phenomena involving two heterogeneous system states. Such phenomena may arise in biology or communications, among other fields. Namely, we consider that a sequence of meaningful words is to be searched within a whole observation that also contains arbitrary one-by-one symbols. Moreover, a word may be interrupted at some site to be carried on later. Applying plain hidden Markov chains to such data, while ignoring their specificity, yields unsatisfactory results. The Phasic triplet Markov chain, proposed in this paper, overcomes this difficulty by means of an auxiliary underlying process in accordance with the triplet Markov chains theory. Related Bayesian restoration techniques and parameters estimation procedures according to the new model are then described. Finally, to assess the performance of the proposed model against the conventional hidden Markov chain model, experiments are conducted on synthetic and real data.

  17. Signal Processing of MEMS Gyroscope Arrays to Improve Accuracy Using a 1st Order Markov for Rate Signal Modeling

    Directory of Open Access Journals (Sweden)

    Weizheng Yuan

    2012-02-01

    Full Text Available This paper presents a signal processing technique to improve angular rate accuracy of the gyroscope by combining the outputs of an array of MEMS gyroscope. A mathematical model for the accuracy improvement was described and a Kalman filter (KF was designed to obtain optimal rate estimates. Especially, the rate signal was modeled by a first-order Markov process instead of a random walk to improve overall performance. The accuracy of the combined rate signal and affecting factors were analyzed using a steady-state covariance. A system comprising a six-gyroscope array was developed to test the presented KF. Experimental tests proved that the presented model was effective at improving the gyroscope accuracy. The experimental results indicated that six identical gyroscopes with an ARW noise of 6.2 °/√h and a bias drift of 54.14 °/h could be combined into a rate signal with an ARW noise of 1.8 °/√h and a bias drift of 16.3 °/h, while the estimated rate signal by the random walk model has an ARW noise of 2.4 °/√h and a bias drift of 20.6 °/h. It revealed that both models could improve the angular rate accuracy and have a similar performance in static condition. In dynamic condition, the test results showed that the first-order Markov process model could reduce the dynamic errors 20% more than the random walk model.

  18. Signal Processing of MEMS Gyroscope Arrays to Improve Accuracy Using a 1st Order Markov for Rate Signal Modeling

    Science.gov (United States)

    Jiang, Chengyu; Xue, Liang; Chang, Honglong; Yuan, Guangmin; Yuan, Weizheng

    2012-01-01

    This paper presents a signal processing technique to improve angular rate accuracy of the gyroscope by combining the outputs of an array of MEMS gyroscope. A mathematical model for the accuracy improvement was described and a Kalman filter (KF) was designed to obtain optimal rate estimates. Especially, the rate signal was modeled by a first-order Markov process instead of a random walk to improve overall performance. The accuracy of the combined rate signal and affecting factors were analyzed using a steady-state covariance. A system comprising a six-gyroscope array was developed to test the presented KF. Experimental tests proved that the presented model was effective at improving the gyroscope accuracy. The experimental results indicated that six identical gyroscopes with an ARW noise of 6.2 °/√h and a bias drift of 54.14 °/h could be combined into a rate signal with an ARW noise of 1.8 °/√h and a bias drift of 16.3 °/h, while the estimated rate signal by the random walk model has an ARW noise of 2.4 °/√h and a bias drift of 20.6 °/h. It revealed that both models could improve the angular rate accuracy and have a similar performance in static condition. In dynamic condition, the test results showed that the first-order Markov process model could reduce the dynamic errors 20% more than the random walk model. PMID:22438734

  19. [Non-equilibrium thermodynamic separation theory of nonlinear chromatography. II. The 0-1 model for nonlinear-mass transfer kinetic processes].

    Science.gov (United States)

    Liang, Heng; Jia, Zhenbin

    2007-11-01

    In the optimal design and control of preparative chromatographic processes, the obstacles appear when one tries to link the Wilson' s framework of chromatographic theories based on partial differential equations (PDEs) with the Eulerian presentation to optimal control approaches based on discrete time states, such as Markov decision processes (MDP) or Model predictive control (MPC). In this paper, the 0-1 model is presented to overcome the obstacles for nonlinear transport chromatography (NTC). With the Lagrangian-Eulerian description (L-ED), one solute cell unit is split into two solute cells, one (SCm) in the mobile phase with the linear velocity of the mobile phase, and the other (SCs) in the stationary phase with zero-velocity. The thermodynamic state vector, S(k), which comprises four vector components, i.e., the sequence number, the position and the local solute concentrations in both SCms and SCses, is introduced to describe the local thermodynamic path (LTP) and the macroscopical thermodynamic path (MTP). For the NTC, the LTP is designed for a solute zone to evolve from the state, S(k), to the virtual migration state, S(M), undergoing the virtual net migration sub-process, and then to the state, S(k+1), undergoing the virtual net inter phase mass transfer sub-process in a short time interval. Complete thermodynamic state iterations with the Markov characteristics are derived by using the local equilibrium isotherm and the local lumped mass transfer coefficient. When the local thermodynamic equilibrium is retained, excellent properties, such as consistency, stability, conservation, accuracy, etc., of the numerical solution of the 0-1 model are observed in the theoretical analysis and in the numerical experiments of the nonlinear ideal chromatography. It is found that the 0-1 model could properly link up with the MDP or optimal control approaches based on discrete time states.

  20. NONLINEAR MODEL PREDICTIVE CONTROL OF CHEMICAL PROCESSES

    Directory of Open Access Journals (Sweden)

    R. G. SILVA

    1999-03-01

    Full Text Available A new algorithm for model predictive control is presented. The algorithm utilizes a simultaneous solution and optimization strategy to solve the model's differential equations. The equations are discretized by equidistant collocation, and along with the algebraic model equations are included as constraints in a nonlinear programming (NLP problem. This algorithm is compared with the algorithm that uses orthogonal collocation on finite elements. The equidistant collocation algorithm results in simpler equations, providing a decrease in computation time for the control moves. Simulation results are presented and show a satisfactory performance of this algorithm.

  1. Coupled parametric processes in binary nonlinear photonic structures

    CERN Document Server

    Saygin, M Yu

    2016-01-01

    We study parametric interactions in a new type of nonlinear photonic structures, which is realized in the vicinity of a pair of nonlinear crystals. In this kind of structure, which we call binary, multiple nonlinear optical processes can be implemented simultaneously, owing to multiple phase-matching conditions, fulfilled separately in the constituent crystals. The coupling between the nonlinear processes by means of modes sharing similar frequency is attained by the spatially-broadband nature of the parametric fields. We investigate the spatial properties of the fields generated in the binary structure constructed from periodically poled crystals for the two examples: 1) single parametric down-conversion, and 2) coupled parametric down-conversion and up-conversion processes. The efficacy of the fields' generation in these examples is analyzed through comparison with the cases of traditional single periodically poled crystal and aperiodic photonic structure, respectively. It has been shown that the relative s...

  2. NONLINEAR OPTICS: Nonlinear optical processes in planar waveguides and excitation of surface polaritons

    Science.gov (United States)

    Yashkir, O. V.; Yashkir, Yu N.

    1987-11-01

    An investigation is made of nonlinear optical interaction of light propagating in a planar waveguide with surface polaritons. Reduced wave equations for the amplitudes of the waveguide modes and surface polaritons are used to study the characteristics of generation of surface polaritons of difference frequency, parametric frequency up-conversion of the polaritons, and stimulated Raman scattering by the polaritons. An analysis is made of the characteristic properties of the investigated nonlinear optical processes.

  3. Si-rich Silicon Nitride for Nonlinear Signal Processing Applications.

    Science.gov (United States)

    Lacava, Cosimo; Stankovic, Stevan; Khokhar, Ali Z; Bucio, T Dominguez; Gardes, F Y; Reed, Graham T; Richardson, David J; Petropoulos, Periklis

    2017-02-02

    Nonlinear silicon photonic devices have attracted considerable attention thanks to their ability to show large third-order nonlinear effects at moderate power levels allowing for all-optical signal processing functionalities in miniaturized components. Although significant efforts have been made and many nonlinear optical functions have already been demonstrated in this platform, the performance of nonlinear silicon photonic devices remains fundamentally limited at the telecom wavelength region due to the two photon absorption (TPA) and related effects. In this work, we propose an alternative CMOS-compatible platform, based on silicon-rich silicon nitride that can overcome this limitation. By carefully selecting the material deposition parameters, we show that both of the device linear and nonlinear properties can be tuned in order to exhibit the desired behaviour at the selected wavelength region. A rigorous and systematic fabrication and characterization campaign of different material compositions is presented, enabling us to demonstrate TPA-free CMOS-compatible waveguides with low linear loss (~1.5 dB/cm) and enhanced Kerr nonlinear response (Re{γ} = 16 Wm(-1)). Thanks to these properties, our nonlinear waveguides are able to produce a π nonlinear phase shift, paving the way for the development of practical devices for future optical communication applications.

  4. Age-related alterations of relaxation processes and non-Markov effects in stochastic dynamics of R-R intervals variability from human ECGs

    CERN Document Server

    Yulmetyev, R M; Panischev, O Y; Hänggi, P; Yulmetyev, Renat M.; Demin, Sergey A.; Panischev, Oleg Yu.; H\\"anggi, Peter

    2005-01-01

    In this paper we consider the age-related alterations of heart rate variability on the basis of the study of non-Markovian effects. The age dynamics of relaxation processes is quantitatively described by means of local relaxation parameters, calculated by the specific localization procedure. We offer a quantitative informational measure of non-Markovity to evaluate the change of statistical effects of memory. Local relaxation parameters for young and elderly people differ by 3.3 times, and quantitative measures of non-Markovity differ by 4.2 times. The comparison of quantitative parameters allows to draw conclusions about the reduction of relaxation rate with ageing and the higher degree of the Markovity of heart rate variability of elderly people.

  5. Saturation process of nonlinear standing waves

    Institute of Scientific and Technical Information of China (English)

    马大猷; 刘克

    1996-01-01

    The sound pressure of the nonlinear standing waves is distorted as expected, but also tends to saturate as being found in standing-wave tube experiments with increasing sinusoidal excitation. Saturation conditions were not actually reached, owing to limited excitation power, but the evidence of tendency to saturation is without question. It is the purpose of this investigation to find the law of saturation from the existing experimental data. The results of curve fitting indicate that negative feedback limits the growth of sound pressure with increasing excitation, the growth of the fundamental and the second harmonic by the negative feedback of their sound pressures, and the growth of the third and higher harmonics, however, by their energies (sound pressures squared). The growth functions of all the harmonics are derived, which are confirmed by the experiments. The saturation pressures and their properties are found.

  6. A Markov decision process for managing habitat for Florida scrub-jays

    Science.gov (United States)

    Johnson, Fred A.; Breininger, David R.; Duncan, Brean W.; Nichols, James D.; Runge, Michael C.; Williams, B. Ken

    2011-01-01

    Florida scrub-jays Aphelocoma coerulescens are listed as threatened under the Endangered Species Act due to loss and degradation of scrub habitat. This study concerned the development of an optimal strategy for the restoration and management of scrub habitat at Merritt Island National Wildlife Refuge, which contains one of the few remaining large populations of scrub-jays in Florida. There are documented differences in the reproductive and survival rates of scrubjays among discrete classes of scrub height (Markov models to estimate annual transition probabilities among the four scrub-height classes under three possible management actions: scrub restoration (mechanical cutting followed by burning), a prescribed burn, or no intervention. A strategy prescribing the optimal management action for management units exhibiting different proportions of scrub-height classes was derived using dynamic programming. Scrub restoration was the optimal management action only in units dominated by mixed and tall scrub, and burning tended to be the optimal action for intermediate levels of short scrub. The optimal action was to do nothing when the amount of short scrub was greater than 30%, because short scrub mostly transitions to optimal height scrub (i.e., that state with the highest demographic success of scrub-jays) in the absence of intervention. Monte Carlo simulation of the optimal policy suggested that some form of management would be required every year. We note, however, that estimates of scrub-height transition probabilities were subject to several sources of uncertainty, and so we explored the management implications of alternative sets of transition probabilities. Generally, our analysis demonstrated the difficulty of managing for a species that requires midsuccessional habitat, and suggests that innovative management tools may be needed to help ensure the persistence of scrub-jays at Merritt Island National Wildlife Refuge. The development of a tailored monitoring

  7. Using model-based proposals for fast parameter inference on discrete state space, continuous-time Markov processes.

    Science.gov (United States)

    Pooley, C M; Bishop, S C; Marion, G

    2015-06-06

    Bayesian statistics provides a framework for the integration of dynamic models with incomplete data to enable inference of model parameters and unobserved aspects of the system under study. An important class of dynamic models is discrete state space, continuous-time Markov processes (DCTMPs). Simulated via the Doob-Gillespie algorithm, these have been used to model systems ranging from chemistry to ecology to epidemiology. A new type of proposal, termed 'model-based proposal' (MBP), is developed for the efficient implementation of Bayesian inference in DCTMPs using Markov chain Monte Carlo (MCMC). This new method, which in principle can be applied to any DCTMP, is compared (using simple epidemiological SIS and SIR models as easy to follow exemplars) to a standard MCMC approach and a recently proposed particle MCMC (PMCMC) technique. When measurements are made on a single-state variable (e.g. the number of infected individuals in a population during an epidemic), model-based proposal MCMC (MBP-MCMC) is marginally faster than PMCMC (by a factor of 2-8 for the tests performed), and significantly faster than the standard MCMC scheme (by a factor of 400 at least). However, when model complexity increases and measurements are made on more than one state variable (e.g. simultaneously on the number of infected individuals in spatially separated subpopulations), MBP-MCMC is significantly faster than PMCMC (more than 100-fold for just four subpopulations) and this difference becomes increasingly large. © 2015 The Author(s) Published by the Royal Society. All rights reserved.

  8. Lifetime effectiveness of mifamurtide addition to chemotherapy in nonmetastatic and metastatic osteosarcoma: a Markov process model analysis.

    Science.gov (United States)

    Song, Hyun Jin; Lee, Jun Ah; Han, Euna; Lee, Eui-Kyung

    2015-09-01

    The mortality and progression rates in osteosarcoma differ depending on the presence of metastasis. A decision model would be useful for estimating long-term effectiveness of treatment with limited clinical trial data. The aim of this study was to explore the lifetime effectiveness of the addition of mifamurtide to chemotherapy for patients with metastatic and nonmetastatic osteosarcoma. The target population was osteosarcoma patients with or without metastasis. A Markov process model was used, whose time horizon was lifetime with a starting age of 13 years. There were five health states: disease-free (DF), recurrence, post-recurrence disease-free, post-recurrence disease-progression, and death. Transition probabilities of the starting state, DF, were calculated from the INT-0133 clinical trials for chemotherapy with and without mifamurtide. Quality-adjusted life-years (QALY) increased upon addition of mifamurtide to chemotherapy by 10.5 % (10.13 and 9.17 QALY with and without mifamurtide, respectively) and 45.2 % (7.23 and 4.98 QALY with and without mifamurtide, respectively) relative to the lifetime effectiveness of chemotherapy in nonmetastatic and metastatic osteosarcoma, respectively. Life-years gained (LYG) increased by 10.1 % (13.10 LYG with mifamurtide and 11.90 LYG without mifamurtide) in nonmetastatic patients and 42.2 % (9.43 LYG with mifamurtide and 6.63 LYG without mifamurtide) in metastatic osteosarcoma patients. The Markov model analysis showed that chemotherapy with mifamurtide improved the lifetime effectiveness compared to chemotherapy alone in both nonmetastatic and metastatic osteosarcoma. Relative effectiveness of the therapy was higher in metastatic than nonmetastatic osteosarcoma over lifetime. However, absolute lifetime effectiveness was higher in nonmetastatic than metastatic osteosarcoma.

  9. Adaptive control method for nonlinear time-delay processes

    Institute of Scientific and Technical Information of China (English)

    2007-01-01

    Two complex properties,varying time-delay and block-oriented nonlinearity,are very common in chemical engineering processes and not easy to be controlled by routine control methods.Aimed at these two complex properties,a novel adaptive control algorithm the basis of nonlinear OFS(orthonormal functional series) model is proposed.First,the hybrid model which combines OFS and Volterra series is introduced.Then,a stable state feedback strategy is used to construct a nonlinear adaptive control algorithm that can guarantee the closed-loop stability and can track the set point curve without steady-state errors.Finally,control simulations and experiments on a nonlinear process with varying time-delay are presented.A number of experimental results validate the efficiency and superiority of this algorithm.

  10. Classification of Markov chains describing the evolution of random strings

    Science.gov (United States)

    Gairat, A. S.; Malyshev, V. A.; Men'shikov, M. V.; Pelikh, K. D.

    1995-04-01

    Contents §1. Introduction §2. Linear equation for average times: the case d = 1 Associated branching process The simplest queue §3. Fundamental non-linear equation: the case d > 1Minimal solution of the equation F(p) = p §4. The main criterion. The case d > 1 §5. Classification of Markov chains by means of a minimal solution §6. Appendix Bibliography

  11. Nonlinear fiber applications for ultrafast all-optical signal processing

    Science.gov (United States)

    Kravtsov, Konstantin

    In the present dissertation different aspects of all-optical signal processing, enabled by the use of nonlinear fibers, are studied. In particular, we focus on applications of a novel heavily GeO2-doped (HD) nonlinear fiber, that appears to be superior to many other types of nonlinear fibers because of its high nonlinearity and suitability for the use in nonlinear optical loop mirrors (NOLMs). Different functions, such as all-optical switching, thresholding, and wavelength conversion, are demonstrated with the HD fibers in the NOLM configuration. These basic functions are later used for realization of ultrafast time-domain demultiplexers, clock recovery, detectors of short pulses in stealth communications, and primitive elements for analog computations. Another important technology that benefits from the use of nonlinear fiber-based signal processing is optical code-division multiple access (CDMA). It is shown in both theory and experiment that all-optical thresholding is a unique way of improving existing detection methods for optical CDMA. Also, it is the way of implementation of true asynchronous optical spread-spectrum networks, which allows full realization of optical CDMA potential. Some aspects of quantum signal processing and manipulation of quantum states are also studied in this work. It is shown that propagation and collisions of Thirring solitons lead to a substantial squeezing of quantum states, which may find applications for generation of squeezed light.

  12. Graphs: Associated Markov Chains

    OpenAIRE

    2012-01-01

    In this research paper, weighted / unweighted, directed / undirected graphs are associated with interesting Discrete Time Markov Chains (DTMCs) as well as Continuous Time Markov Chains (CTMCs). The equilibrium / transient behaviour of such Markov chains is studied. Also entropy dynamics (Shannon entropy) of certain structured Markov chains is investigated. Finally certain structured graphs and the associated Markov chains are studied.

  13. Nonlinear Statistical Process Monitoring and Fault Detection Using Kernel ICA

    Institute of Scientific and Technical Information of China (English)

    ZHANG Xi; YAN Wei-wu; ZHAO Xu; SHAO Hui-he

    2007-01-01

    A novel nonlinear process monitoring and fault detection method based on kernel independent component analysis (ICA) is proposed. The kernel ICA method is a two-phase algorithm: whitened kernel principal component (KPCA) plus ICA. KPCA spheres data and makes the data structure become as linearly separable as possible by virtue of an implicit nonlinear mapping determined by kernel. ICA seeks the projection directions in the KPCA whitened space, making the distribution of the projected data as non-gaussian as possible. The application to the fluid catalytic cracking unit (FCCU) simulated process indicates that the proposed process monitoring method based on kernel ICA can effectively capture the nonlinear relationship in process variables. Its performance significantly outperforms monitoring method based on ICA or KPCA.

  14. Modeling and stability analysis of the nonlinear reactive sputtering process

    Directory of Open Access Journals (Sweden)

    György Katalin

    2011-12-01

    Full Text Available The model of the reactive sputtering process has been determined from the dynamic equilibrium of the reactive gas inside the chamber and the dynamic equilibrium of the sputtered metal atoms which form the compound with the reactive gas atoms on the surface of the substrate. The analytically obtained dynamical model is a system of nonlinear differential equations which can result in a histeresis-type input/output nonlinearity. The reactive sputtering process has been simulated by integrating these differential equations. Linearization has been applied for classical analysis of the sputtering process and control system design.

  15. An Agent Interaction Based Method for Nonlinear Process Plan Scheduling

    Institute of Scientific and Technical Information of China (English)

    GAO Qinglu; WU Bo; GUO Guang

    2006-01-01

    This article puts forward a scheduling method for nonlinear process plan shop floor. Task allocation and load balance are realized by bidding mechanism. Though the agent interaction process, the execution of tasks is determined and the coherence of manufacturing decision is verified. The employment of heuristic index can help to optimize the system performance.

  16. Innovation as a Nonlinear Process and the Scientometric Perspective

    NARCIS (Netherlands)

    Leydesdorff, L.; Rotolo, D.; de Nooy, W.; Archambault, E.; Gingras, Y.; Larivière, V.

    2012-01-01

    The process of innovation follows non-linear patterns across the domains of science, technology, and the economy. Novel bibliometric mapping techniques can be used to investigate and represent distinctive, but complementary perspectives on the innovation process (e.g., "demand" and "supply") as well

  17. Stochastic nonlinear differential equations. I

    NARCIS (Netherlands)

    Heilmann, O.J.; Kampen, N.G. van

    1974-01-01

    A solution method is developed for nonlinear differential equations having the following two properties. Their coefficients are stochastic through their dependence on a Markov process. The magnitude of the fluctuations, multiplied with their auto-correlation time, is a small quantity. Under these co

  18. Two-boundary first exit time of Gauss-Markov processes for stochastic modeling of acto-myosin dynamics.

    Science.gov (United States)

    D'Onofrio, Giuseppe; Pirozzi, Enrica

    2017-05-01

    We consider a stochastic differential equation in a strip, with coefficients suitably chosen to describe the acto-myosin interaction subject to time-varying forces. By simulating trajectories of the stochastic dynamics via an Euler discretization-based algorithm, we fit experimental data and determine the values of involved parameters. The steps of the myosin are represented by the exit events from the strip. Motivated by these results, we propose a specific stochastic model based on the corresponding time-inhomogeneous Gauss-Markov and diffusion process evolving between two absorbing boundaries. We specify the mean and covariance functions of the stochastic modeling process taking into account time-dependent forces including the effect of an external load. We accurately determine the probability density function (pdf) of the first exit time (FET) from the strip by solving a system of two non singular second-type Volterra integral equations via a numerical quadrature. We provide numerical estimations of the mean of FET as approximations of the dwell-time of the proteins dynamics. The percentage of backward steps is given in agreement to experimental data. Numerical and simulation results are compared and discussed.

  19. Markov Chain Monte Carlo Random Effects Modeling in Magnetic Resonance Image Processing Using the BRugs Interface to WinBUGS

    Directory of Open Access Journals (Sweden)

    David G. Gadian

    2011-10-01

    Full Text Available A common feature of many magnetic resonance image (MRI data processing methods is the voxel-by-voxel (a voxel is a volume element manner in which the processing is performed. In general, however, MRI data are expected to exhibit some level of spatial correlation, rendering an independent-voxels treatment inefficient in its use of the data. Bayesian random effect models are expected to be more efficient owing to their information-borrowing behaviour. To illustrate the Bayesian random effects approach, this paper outlines a Markov chain Monte Carlo (MCMC analysis of a perfusion MRI dataset, implemented in R using the BRugs package. BRugs provides an interface to WinBUGS and its GeoBUGS add-on. WinBUGS is a widely used programme for performing MCMC analyses, with a focus on Bayesian random effect models. A simultaneous modeling of both voxels (restricted to a region of interest and multiple subjects is demonstrated. Despite the low signal-to-noise ratio in the magnetic resonance signal intensity data, useful model signal intensity profiles are obtained. The merits of random effects modeling are discussed in comparison with the alternative approaches based on region-of-interest averaging and repeated independent voxels analysis. This paper focuses on perfusion MRI for the purpose of illustration, the main proposition being that random effects modeling is expected to be beneficial in many other MRI applications in which the signal-to-noise ratio is a limiting factor.

  20. Binary hidden Markov models and varieties

    CERN Document Server

    Critch, Andrew J

    2012-01-01

    The technological applications of hidden Markov models have been extremely diverse and successful, including natural language processing, gesture recognition, gene sequencing, and Kalman filtering of physical measurements. HMMs are highly non-linear statistical models, and just as linear models are amenable to linear algebraic techniques, non-linear models are amenable to commutative algebra and algebraic geometry. This paper examines closely those HMMs in which all the random variables, called nodes, are binary. Its main contributions are (1) minimal defining equations for the 4-node model, comprising 21 quadrics and 29 cubics, which were computed using Gr\\"obner bases in the cumulant coordinates of Sturmfels and Zwiernik, and (2) a birational parametrization for every binary HMM, with an explicit inverse for recovering the hidden parameters in terms of observables. The new model parameters in (2) are hence rationally identifiable in the sense of Sullivant, Garcia-Puente, and Spielvogel, and each model's Zar...

  1. Markov chains theory and applications

    CERN Document Server

    Sericola, Bruno

    2013-01-01

    Markov chains are a fundamental class of stochastic processes. They are widely used to solve problems in a large number of domains such as operational research, computer science, communication networks and manufacturing systems. The success of Markov chains is mainly due to their simplicity of use, the large number of available theoretical results and the quality of algorithms developed for the numerical evaluation of many metrics of interest.The author presents the theory of both discrete-time and continuous-time homogeneous Markov chains. He carefully examines the explosion phenomenon, the

  2. Nonlinear partial least squares with Hellinger distance for nonlinear process monitoring

    KAUST Repository

    Harrou, Fouzi

    2017-02-16

    This paper proposes an efficient data-based anomaly detection method that can be used for monitoring nonlinear processes. The proposed method merges advantages of nonlinear projection to latent structures (NLPLS) modeling and those of Hellinger distance (HD) metric to identify abnormal changes in highly correlated multivariate data. Specifically, the HD is used to quantify the dissimilarity between current NLPLS-based residual and reference probability distributions. The performances of the developed anomaly detection using NLPLS-based HD technique is illustrated using simulated plug flow reactor data.

  3. Evaluation Of The Safety Integrity Level (SIL Due To The Guidelines Of EN 61508 And With The Use Of Markov Processes

    Directory of Open Access Journals (Sweden)

    Młynarski Stanisław

    2015-11-01

    Full Text Available The methods of evaluation of the Probability of Failure on Demand (PFD of safety systems were presented in the paper, assuming that the safety systems may be represented by the k out of n reliability structures. The results of the calculations obtained according to EN 61508 were compared with another results, this time obtained from the calculations done for these systems assuming that their failure-and-renewal process is a Markov process.

  4. Nonlinear Dynamic Characteristics of Combustion Wave in SHS Process

    Institute of Scientific and Technical Information of China (English)

    2002-01-01

    The characteristic of combustion wave and its change were analyzed by numerical value calculation and computer simulation,based on the combustion dynamical model of SHS process. It is shown that with the change of condition parameters in SHS process various time-space order combustion waves appear.It is concluded from non-liner dynamical mechanism analysis that the strong coupling of two non-linear dynamical processes is the dynamical mechanism causing the time-space order dissipation structures.

  5. Relaxation Processes in Nonlinear Optical Polymer Films

    Directory of Open Access Journals (Sweden)

    S.N. Fedosov

    2010-01-01

    Full Text Available Dielectric properties of the guest-host polystyrene/DR1 system have been studied by the AC dielectric spectroscopy method at frequencies from 1 Hz to 0,5 MHz and by the thermally stimulated depolarization current (TSDC method from – 160 to 0 °C. The relaxation peaks at infra-low frequencies from 10 – 5to 10–2 Hz were also calculated using the Hamon’s approximation. Three relaxation processes, namely, α, β and δ ones were identified from the TSDC peaks, while the ε''(fdependence showed a non-Debye ρ-peak narrowing with temperature. The activation energy of the α-relaxation appeared to be 2,57 eV, while that of the γ-process was 0,52 eV. Temperature dependence of the relaxation time is agreed with the Williams-Landel-Ferry model. The ε''(fpeaks were fitted to Havriliak-Negami’s expression and the corresponding distribution parameters were obtained.

  6. Entropy: The Markov Ordering Approach

    Directory of Open Access Journals (Sweden)

    Alexander N. Gorban

    2010-05-01

    Full Text Available The focus of this article is on entropy and Markov processes. We study the properties of functionals which are invariant with respect to monotonic transformations and analyze two invariant “additivity” properties: (i existence of a monotonic transformation which makes the functional additive with respect to the joining of independent systems and (ii existence of a monotonic transformation which makes the functional additive with respect to the partitioning of the space of states. All Lyapunov functionals for Markov chains which have properties (i and (ii are derived. We describe the most general ordering of the distribution space, with respect to which all continuous-time Markov processes are monotonic (the Markov order. The solution differs significantly from the ordering given by the inequality of entropy growth. For inference, this approach results in a convex compact set of conditionally “most random” distributions.

  7. Multimodal Brain-Tumor Segmentation Based on Dirichlet Process Mixture Model with Anisotropic Diffusion and Markov Random Field Prior

    Directory of Open Access Journals (Sweden)

    Yisu Lu

    2014-01-01

    Full Text Available Brain-tumor segmentation is an important clinical requirement for brain-tumor diagnosis and radiotherapy planning. It is well-known that the number of clusters is one of the most important parameters for automatic segmentation. However, it is difficult to define owing to the high diversity in appearance of tumor tissue among different patients and the ambiguous boundaries of lesions. In this study, a nonparametric mixture of Dirichlet process (MDP model is applied to segment the tumor images, and the MDP segmentation can be performed without the initialization of the number of clusters. Because the classical MDP segmentation cannot be applied for real-time diagnosis, a new nonparametric segmentation algorithm combined with anisotropic diffusion and a Markov random field (MRF smooth constraint is proposed in this study. Besides the segmentation of single modal brain-tumor images, we developed the algorithm to segment multimodal brain-tumor images by the magnetic resonance (MR multimodal features and obtain the active tumor and edema in the same time. The proposed algorithm is evaluated using 32 multimodal MR glioma image sequences, and the segmentation results are compared with other approaches. The accuracy and computation time of our algorithm demonstrates very impressive performance and has a great potential for practical real-time clinical use.

  8. Markov Decision Process Based Energy-Efficient On-Line Scheduling for Slice-Parallel Video Decoders on Multicore Systems

    CERN Document Server

    Mastronarde, Nicholas; Atienza, David; Frossard, Pascal; van der Schaar, Mihaela

    2011-01-01

    We consider the problem of energy-efficient on-line scheduling for slice-parallel video decoders on multicore systems. We assume that each of the processors are Dynamic Voltage Frequency Scaling (DVFS) enabled such that they can independently trade off performance for power, while taking the video decoding workload into account. In the past, scheduling and DVFS policies in multi-core systems have been formulated heuristically due to the inherent complexity of the on-line multicore scheduling problem. The key contribution of this report is that we rigorously formulate the problem as a Markov decision process (MDP), which simultaneously takes into account the on-line scheduling and per-core DVFS capabilities; the separate power consumption of the processor cores and caches; and the loss tolerant and dynamic nature of the video decoder's traffic. In particular, we model the video traffic using a Direct Acyclic Graph (DAG) to capture the precedence constraints among frames in a Group of Pictures (GOP) structure, ...

  9. Markov chains

    CERN Document Server

    Revuz, D

    1984-01-01

    This is the revised and augmented edition of a now classic book which is an introduction to sub-Markovian kernels on general measurable spaces and their associated homogeneous Markov chains. The first part, an expository text on the foundations of the subject, is intended for post-graduate students. A study of potential theory, the basic classification of chains according to their asymptotic behaviour and the celebrated Chacon-Ornstein theorem are examined in detail. The second part of the book is at a more advanced level and includes a treatment of random walks on general locally compact abelian groups. Further chapters develop renewal theory, an introduction to Martin boundary and the study of chains recurrent in the Harris sense. Finally, the last chapter deals with the construction of chains starting from a kernel satisfying some kind of maximum principle.

  10. REPRESENTATION OF ADDITIVE FUNCTIONALS AND LOCAL TIMES FOR JUMP MARKOV PROCESSES AND THEIR FUNCTIONAL LIMIT THEOREM

    Institute of Scientific and Technical Information of China (English)

    蒋义文; 刘禄勤

    2003-01-01

    The representation of additive functionals and local times for jump Markovprocesses are obtained. The results of uniformly functional moderate deviation and theirapplications to birth-death processes are also presented.

  11. A Markov Decision Process Model for the Optimal Dispatch of Military Medical Evacuation Assets

    Science.gov (United States)

    2014-03-27

    coalition forces in Afghanistan closely resemble crime patterns, they can be analyzed as a contagion -like process. The Hawkes spatial generation process (see...Master of Science (Operations Research ) Sean K. Keneally, BS, MS Major, USA March 2014 DISTRIBUTION STATEMENT A. APPROVED FOR PUBLIC RELEASE...indicated by the 9-line MEDEVAC request. The proposed MDP model indicates how to optimally dispatch ambulatory helicopters to casualty events in order to

  12. Ultra-Fast Optical Signal Processing in Nonlinear Silicon Waveguides

    DEFF Research Database (Denmark)

    Oxenløwe, Leif Katsuo; Galili, Michael; Pu, Minhao;

    2011-01-01

    We describe recent demonstrations of exploiting highly nonlinear silicon nanowires for processing Tbit/s optical data signals. We perform demultiplexing and optical waveform sampling of 1.28 Tbit/s and wavelength conversion of 640 Gbit/s data signals....

  13. Nonlinear Evolutions of Stimulated Raman and Brillouin Scattering Processes in Partially Stripped-Ion Plasmas

    Institute of Scientific and Technical Information of China (English)

    胡业民; 胡希伟

    2001-01-01

    Numerical analyses for the nonlinear evolutions of stimulated Raman scattering (SRS) and stimulated Brillouin scattering (SBS) processes are given. Various effects of the second- and third-order nonlinear susceptibilities on the SRS and SBS processes are studied. The nonlinear evolutions of SRS and SBS processes are atfected more efficiently than their linear growth rates by the nonlinear susceptibility.

  14. Markov chain Monte Carlo simulation for Bayesian Hidden Markov Models

    Science.gov (United States)

    Chan, Lay Guat; Ibrahim, Adriana Irawati Nur Binti

    2016-10-01

    A hidden Markov model (HMM) is a mixture model which has a Markov chain with finite states as its mixing distribution. HMMs have been applied to a variety of fields, such as speech and face recognitions. The main purpose of this study is to investigate the Bayesian approach to HMMs. Using this approach, we can simulate from the parameters' posterior distribution using some Markov chain Monte Carlo (MCMC) sampling methods. HMMs seem to be useful, but there are some limitations. Therefore, by using the Mixture of Dirichlet processes Hidden Markov Model (MDPHMM) based on Yau et. al (2011), we hope to overcome these limitations. We shall conduct a simulation study using MCMC methods to investigate the performance of this model.

  15. Generalized Markov branching models

    OpenAIRE

    Li, Junping

    2005-01-01

    In this thesis, we first considered a modified Markov branching process incorporating both state-independent immigration and resurrection. After establishing the criteria for regularity and uniqueness, explicit expressions for the extinction probability and mean extinction time are presented. The criteria for recurrence and ergodicity are also established. In addition, an explicit expression for the equilibrium distribution is presented.\\ud \\ud We then moved on to investigate the basic proper...

  16. Generalized Markov branching models

    OpenAIRE

    Li, Junping

    2005-01-01

    In this thesis, we first considered a modified Markov branching process incorporating both state-independent immigration and resurrection. After establishing the criteria for regularity and uniqueness, explicit expressions for the extinction probability and mean extinction time are presented. The criteria for recurrence and ergodicity are also established. In addition, an explicit expression for the equilibrium distribution is presented. We then moved on to investigate the basic proper...

  17. Modeling Autoregressive Processes with Moving-Quantiles-Implied Nonlinearity

    Directory of Open Access Journals (Sweden)

    Isao Ishida

    2015-01-01

    Full Text Available We introduce and investigate some properties of a class of nonlinear time series models based on the moving sample quantiles in the autoregressive data generating process. We derive a test fit to detect this type of nonlinearity. Using the daily realized volatility data of Standard & Poor’s 500 (S&P 500 and several other indices, we obtained good performance using these models in an out-of-sample forecasting exercise compared with the forecasts obtained based on the usual linear heterogeneous autoregressive and other models of realized volatility.

  18. Improved object segmentation using Markov random fields, artificial neural networks, and parallel processing techniques

    Science.gov (United States)

    Foulkes, Stephen B.; Booth, David M.

    1997-07-01

    Object segmentation is the process by which a mask is generated which identifies the area of an image which is occupied by an object. Many object recognition techniques depend on the quality of such masks for shape and underlying brightness information, however, segmentation remains notoriously unreliable. This paper considers how the image restoration technique of Geman and Geman can be applied to the improvement of object segmentations generated by a locally adaptive background subtraction technique. Also presented is how an artificial neural network hybrid, consisting of a single layer Kohonen network with each of its nodes connected to a different multi-layer perceptron, can be used to approximate the image restoration process. It is shown that the restoration techniques are very well suited for parallel processing and in particular the artificial neural network hybrid has the potential for near real time image processing. Results are presented for the detection of ships in SPOT panchromatic imagery and the detection of vehicles in infrared linescan images, these being a fair representation of the wider class of problem.

  19. New CMOS Compatible Platforms for Integrated Nonlinear Optical Signal Processing

    CERN Document Server

    Moss, D J

    2014-01-01

    Nonlinear photonic chips have succeeded in generating and processing signals all-optically with performance far superior to that possible electronically - particularly with respect to speed. Although silicon-on-insulator has been the leading platform for nonlinear optics, its high two-photon absorption at telecommunications wavelengths poses a fundamental limitation. This paper reviews some of the recent achievements in CMOS-compatible platforms for nonlinear optics, focusing on amorphous silicon and Hydex glass, highlighting their potential future impact as well as the challenges to achieving practical solutions for many key applications. These material systems have opened up many new capabilities such as on-chip optical frequency comb generation and ultrafast optical pulse generation and measurement.

  20. Linear and Nonlinear MHD Wave Processes in Plasmas. Final Report

    Energy Technology Data Exchange (ETDEWEB)

    Tataronis, J. A.

    2004-06-01

    This program treats theoretically low frequency linear and nonlinear wave processes in magnetized plasmas. A primary objective has been to evaluate the effectiveness of MHD waves to heat plasma and drive current in toroidal configurations. The research covers the following topics: (1) the existence and properties of the MHD continua in plasma equilibria without spatial symmetry; (2) low frequency nonresonant current drive and nonlinear Alfven wave effects; and (3) nonlinear electron acceleration by rf and random plasma waves. Results have contributed to the fundamental knowledge base of MHD activity in symmetric and asymmetric toroidal plasmas. Among the accomplishments of this research effort, the following are highlighted: Identification of the MHD continuum mode singularities in toroidal geometry. Derivation of a third order ordinary differential equation that governs nonlinear current drive in the singular layers of the Alfvkn continuum modes in axisymmetric toroidal geometry. Bounded solutions of this ODE implies a net average current parallel to the toroidal equilibrium magnetic field. Discovery of a new unstable continuum of the linearized MHD equation in axially periodic circular plasma cylinders with shear and incompressibility. This continuum, which we named “accumulation continuum” and which is related to ballooning modes, arises as discrete unstable eigenfrequency accumulate on the imaginary frequency axis in the limit of large mode numbers. Development of techniques to control nonlinear electron acceleration through the action of multiple coherent and random plasmas waves. Two important elements of this program aye student participation and student training in plasma theory.

  1. Process and meaning: nonlinear dynamics and psychology in visual art.

    Science.gov (United States)

    Zausner, Tobi

    2007-01-01

    Creating and viewing visual art are both nonlinear experiences. Creating a work of art is an irreversible process involving increasing levels of complexity and unpredictable events. Viewing art is also creative with collective responses forming autopoietic structures that shape cultural history. Artists work largely from the chaos of the unconscious and visual art contains elements of chaos. Works of art by the author are discussed in reference to nonlinear dynamics. "Travelogues" demonstrates continued emerging interpretations and a deterministic chaos. "Advice to the Imperfect" signifies the resolution of paradox in the nonlinear tension of opposites. "Quanah" shows the nonlinear tension of opposites as an ongoing personal evolution. "The Mother of All Things" depicts seemingly separate phenomena arising from undifferentiated chaos. "Memories" refers to emotional fixations as limit cycles. "Compassionate Heart," "Wind on the Lake," and "Le Mal du Pays" are a series of works in fractal format focusing on the archetype of the mother and child. "Sameness, Depth of Mystery" addresses the illusion of hierarchy and the dynamics of symbols. In "Chasadim" the origin of worlds and the regeneration of individuals emerge through chaos. References to chaos in visual art mirror the nonlinear complexity of life.

  2. Nonlinear Processes in Magnetic Nanodots under Perpendicular Pumping: Micromagnetic Simulations

    Directory of Open Access Journals (Sweden)

    D.V. Slobodiainuk

    2013-03-01

    Full Text Available Processes that take place in permalloy nanodots under external electromagnetic pumping are considered. It is shown that in such system similar to bulk samples Suhl and kinetic instability processes are possible. Using micromagnetic simulations approach key features of mode excitation with an external pumping power increase were revealed. Results of the simulations were compared with published experimental data dedicated to investigation of magnetic nanodotes in nonlinear regime.

  3. Limit Theorems for Functionals of Markov Processes and Renormalizable Stable Fields

    Science.gov (United States)

    1993-01-01

    g(xl,Yl) ... g(xk,yk), 92(a,x) = g2 (a1, x).. .g 2 (ak,-k) (13) (see Feldman and Rachev , 1993), we restrict our parameter set to functions from the...authors would like to thank Svetlozar Rachev for helpful discussions of stable distributions and processes. We would like to thank Makoto Maejima for...limit theorems for functionals of the Brownian sheet, Annals of Probability, 17, 538-558. Feldman, R.E. and Rachev , S.T. (1993), U-statistics of random

  4. Markov过程修正的一则注记%A Note on the Modification of Markov's Stochastic Process

    Institute of Scientific and Technical Information of China (English)

    刘道溥; 邱爱保

    2004-01-01

    随机过程修正广泛应用于研究随机过程,随机流与随机微分系统.在本篇论文中,我们讨论具有Feller半群的Markov 过程的修正,并给了Markov 过程在状态空间S为非紧时修正定理的证明,并应用此修正定理构造了一个Feller过程的例子.

  5. A non-linear model of economic production processes

    Science.gov (United States)

    Ponzi, A.; Yasutomi, A.; Kaneko, K.

    2003-06-01

    We present a new two phase model of economic production processes which is a non-linear dynamical version of von Neumann's neoclassical model of production, including a market price-setting phase as well as a production phase. The rate of an economic production process is observed, for the first time, to depend on the minimum of its input supplies. This creates highly non-linear supply and demand dynamics. By numerical simulation, production networks are shown to become unstable when the ratio of different products to total processes increases. This provides some insight into observed stability of competitive capitalist economies in comparison to monopolistic economies. Capitalist economies are also shown to have low unemployment.

  6. Optoelectronic and nonlinear optical processes in low dimensional semiconductors

    Indian Academy of Sciences (India)

    B P Singh

    2006-11-01

    Spatial confinement of quantum excitations on their characteristic wavelength scale in low dimensional materials offers unique possibilities to engineer the electronic structure and thereby control their physical properties by way of simple manipulation of geometrical parameters. This has led to an overwhelming interest in quasi-zero dimensional semiconductors or quantum dots as tunable materials for multitude of exciting applications in optoelectronic and nonlinear optical devices and quantum information processing. Large nonlinear optical response and high luminescence quantum yield expected in these systems is a consequence of huge enhancement of transition probabilities ensuing from quantum confinement. High quantum efficiency of photoluminescence, however, is not usually realized in the case of bare semiconductor nanoparticles owing to the presence of surface states. In this talk, I will focus on the role of quantum confinement and surface states in ascertaining nonlinear optical and optoelectronic properties of II–VI semiconductor quantum dots and their nanocomposites. I will also discuss the influence of nonlinear optical processes on their optoelectronic characteristics.

  7. Portfolio allocation under the vendor managed inventory: A Markov ...

    African Journals Online (AJOL)

    Portfolio allocation under the vendor managed inventory: A Markov decision process. ... Journal of Applied Sciences and Environmental Management ... Markov decision processes have been applied in solving a wide range of optimization ...

  8. Probabilistic parameter estimation of activated sludge processes using Markov Chain Monte Carlo.

    Science.gov (United States)

    Sharifi, Soroosh; Murthy, Sudhir; Takács, Imre; Massoudieh, Arash

    2014-03-01

    One of the most important challenges in making activated sludge models (ASMs) applicable to design problems is identifying the values of its many stoichiometric and kinetic parameters. When wastewater characteristics data from full-scale biological treatment systems are used for parameter estimation, several sources of uncertainty, including uncertainty in measured data, external forcing (e.g. influent characteristics), and model structural errors influence the value of the estimated parameters. This paper presents a Bayesian hierarchical modeling framework for the probabilistic estimation of activated sludge process parameters. The method provides the joint probability density functions (JPDFs) of stoichiometric and kinetic parameters by updating prior information regarding the parameters obtained from expert knowledge and literature. The method also provides the posterior correlations between the parameters, as well as a measure of sensitivity of the different constituents with respect to the parameters. This information can be used to design experiments to provide higher information content regarding certain parameters. The method is illustrated using the ASM1 model to describe synthetically generated data from a hypothetical biological treatment system. The results indicate that data from full-scale systems can narrow down the ranges of some parameters substantially whereas the amount of information they provide regarding other parameters is small, due to either large correlations between some of the parameters or a lack of sensitivity with respect to the parameters.

  9. The influence of Markov decision process structure on the possible strategic use of working memory and episodic memory.

    Science.gov (United States)

    Zilli, Eric A; Hasselmo, Michael E

    2008-07-23

    Researchers use a variety of behavioral tasks to analyze the effect of biological manipulations on memory function. This research will benefit from a systematic mathematical method for analyzing memory demands in behavioral tasks. In the framework of reinforcement learning theory, these tasks can be mathematically described as partially-observable Markov decision processes. While a wealth of evidence collected over the past 15 years relates the basal ganglia to the reinforcement learning framework, only recently has much attention been paid to including psychological concepts such as working memory or episodic memory in these models. This paper presents an analysis that provides a quantitative description of memory states sufficient for correct choices at specific decision points. Using information from the mathematical structure of the task descriptions, we derive measures that indicate whether working memory (for one or more cues) or episodic memory can provide strategically useful information to an agent. In particular, the analysis determines which observed states must be maintained in or retrieved from memory to perform these specific tasks. We demonstrate the analysis on three simplified tasks as well as eight more complex memory tasks drawn from the animal and human literature (two alternation tasks, two sequence disambiguation tasks, two non-matching tasks, the 2-back task, and the 1-2-AX task). The results of these analyses agree with results from quantitative simulations of the task reported in previous publications and provide simple indications of the memory demands of the tasks which can require far less computation than a full simulation of the task. This may provide a basis for a quantitative behavioral stoichiometry of memory tasks.

  10. The influence of Markov decision process structure on the possible strategic use of working memory and episodic memory.

    Directory of Open Access Journals (Sweden)

    Eric A Zilli

    Full Text Available Researchers use a variety of behavioral tasks to analyze the effect of biological manipulations on memory function. This research will benefit from a systematic mathematical method for analyzing memory demands in behavioral tasks. In the framework of reinforcement learning theory, these tasks can be mathematically described as partially-observable Markov decision processes. While a wealth of evidence collected over the past 15 years relates the basal ganglia to the reinforcement learning framework, only recently has much attention been paid to including psychological concepts such as working memory or episodic memory in these models. This paper presents an analysis that provides a quantitative description of memory states sufficient for correct choices at specific decision points. Using information from the mathematical structure of the task descriptions, we derive measures that indicate whether working memory (for one or more cues or episodic memory can provide strategically useful information to an agent. In particular, the analysis determines which observed states must be maintained in or retrieved from memory to perform these specific tasks. We demonstrate the analysis on three simplified tasks as well as eight more complex memory tasks drawn from the animal and human literature (two alternation tasks, two sequence disambiguation tasks, two non-matching tasks, the 2-back task, and the 1-2-AX task. The results of these analyses agree with results from quantitative simulations of the task reported in previous publications and provide simple indications of the memory demands of the tasks which can require far less computation than a full simulation of the task. This may provide a basis for a quantitative behavioral stoichiometry of memory tasks.

  11. Evaluation of linearly solvable Markov decision process with dynamic model learning in a mobile robot navigation task

    Directory of Open Access Journals (Sweden)

    Ken eKinjo

    2013-04-01

    Full Text Available Linearly solvable Markov Decision Process (LMDP is a class of optimal control problem in whichthe Bellman’s equation can be converted into a linear equation by an exponential transformation ofthe state value function (Todorov, 2009. In an LMDP, the optimal value function and the correspondingcontrol policy are obtained by solving an eigenvalue problem in a discrete state space or an eigenfunctionproblem in a continuous state using the knowledge of the system dynamics and the action, state, andterminal cost functions.In this study, we evaluate the effectiveness of the LMDP framework in real robot control, in whichthe dynamics of the body and the environment have to be learned from experience. We first perform asimulation study of a pole swing-up task to evaluate the effect of the accuracy of the learned dynam-ics model on the derived the action policy. The result shows that a crude linear approximation of thenonlinear dynamics can still allow solution of the task, despite with a higher total cost.We then perform real robot experiments of a battery-catching task using our Spring Dog mobile robotplatform. The state is given by the position and the size of a battery in its camera view and two neck jointangles. The action is the velocities of two wheels, while the neck joints were controlled by a visual servocontroller. We test linear and bilinear dynamic models in tasks with quadratic and Guassian state costfunctions. In the quadratic cost task, the LMDP controller derived from a learned linear dynamics modelperformed equivalently with the optimal linear quadratic controller (LQR. In the non-quadratic task, theLMDP controller with a linear dynamics model showed the best performance. The results demonstratethe usefulness of the LMDP framework in real robot control even when simple linear models are usedfor dynamics learning.

  12. NON-LINEAR FINITE ELEMENT MODELING OF DEEP DRAWING PROCESS

    Directory of Open Access Journals (Sweden)

    Hasan YILDIZ

    2004-03-01

    Full Text Available Deep drawing process is one of the main procedures used in different branches of industry. Finding numerical solutions for determination of the mechanical behaviour of this process will save time and money. In die surfaces, which have complex geometries, it is hard to determine the effects of parameters of sheet metal forming. Some of these parameters are wrinkling, tearing, and determination of the flow of the thin sheet metal in the die and thickness change. However, the most difficult one is determination of material properties during plastic deformation. In this study, the effects of all these parameters are analyzed before producing the dies. The explicit non-linear finite element method is chosen to be used in the analysis. The numerical results obtained for non-linear material and contact models are also compared with the experiments. A good agreement between the numerical and the experimental results is obtained. The results obtained for the models are given in detail.

  13. Nonlinear Silicon Photonic Signal Processing Devices for Future Optical Networks

    Directory of Open Access Journals (Sweden)

    Cosimo Lacava

    2017-01-01

    Full Text Available In this paper, we present a review on silicon-based nonlinear devices for all optical nonlinear processing of complex telecommunication signals. We discuss some recent developments achieved by our research group, through extensive collaborations with academic partners across Europe, on optical signal processing using silicon-germanium and amorphous silicon based waveguides as well as novel materials such as silicon rich silicon nitride and tantalum pentoxide. We review the performance of four wave mixing wavelength conversion applied on complex signals such as Differential Phase Shift Keying (DPSK, Quadrature Phase Shift Keying (QPSK, 16-Quadrature Amplitude Modulation (QAM and 64-QAM that dramatically enhance the telecom signal spectral efficiency, paving the way to next generation terabit all-optical networks.

  14. Preface "Nonlinear processes in oceanic and atmospheric flows"

    Directory of Open Access Journals (Sweden)

    E. García-Ladona

    2010-05-01

    Full Text Available Nonlinear phenomena are essential ingredients in many oceanic and atmospheric processes, and successful understanding of them benefits from multidisciplinary collaboration between oceanographers, meteorologists, physicists and mathematicians. The present Special Issue on "Nonlinear Processes in Oceanic and Atmospheric Flows" contains selected contributions from attendants to the workshop which, in the above spirit, was held in Castro Urdiales, Spain, in July 2008. Here we summarize the Special Issue contributions, which include papers on the characterization of ocean transport in the Lagrangian and in the Eulerian frameworks, generation and variability of jets and waves, interactions of fluid flow with plankton dynamics or heavy drops, scaling in meteorological fields, and statistical properties of El Niño Southern Oscillation.

  15. Preface "Nonlinear processes in oceanic and atmospheric flows"

    CERN Document Server

    Mancho, A M; Turiel, A; Hernandez-Garcia, E; Lopez, C; Garcia-Ladona, E; 10.5194/npg-17-283-2010

    2010-01-01

    Nonlinear phenomena are essential ingredients in many oceanic and atmospheric processes, and successful understanding of them benefits from multidisciplinary collaboration between oceanographers, meteorologists, physicists and mathematicians. The present Special Issue on ``Nonlinear Processes in Oceanic and Atmospheric Flows'' contains selected contributions from attendants to the workshop which, in the above spirit, was held in Castro Urdiales, Spain, in July 2008. Here we summarize the Special Issue contributions, which include papers on the characterization of ocean transport in the Lagrangian and in the Eulerian frameworks, generation and variability of jets and waves, interactions of fluid flow with plankton dynamics or heavy drops, scaling in meteorological fields, and statistical properties of El Ni\\~no Southern Oscillation.

  16. High-speed signal processing using highly nonlinear optical fibres

    DEFF Research Database (Denmark)

    Peucheret, Christophe; Oxenløwe, Leif Katsuo; Mulvad, Hans Christian Hansen

    2009-01-01

    relying on the phase of the optical field. Topics covered include all-optical switching of 640 Gbit/s and 1.28 Tbit/s serial data, wavelength conversion at 640 Gbit/s, optical amplitude regeneration of differential phase shift keying (DPSK) signals, as well as midspan spectral inversion for differential 8......We review recent progress in all-optical signal processing techniques making use of conventional silica-based highly nonlinear fibres. In particular, we focus on recent demonstrations of ultra-fast processing at 640 Gbit/s and above, as well as on signal processing of novel modulation formats...

  17. Double resonant processes in $\\chi^{(2)}$ nonlinear periodic media

    OpenAIRE

    Konotop, V. V.; Kuzmiak, V.

    2000-01-01

    In a one-dimensional periodic nonlinear $\\chi^{(2)}$ medium, by choosing a proper material and geometrical parameters of the structure, it is possible to obtain two matching conditions for simultaneous generation of second and third harmonics. This leads to new diversity of the processes of the resonant three-wave interactions, which are discussed within the framework of slowly varying envelope approach. In particular, we concentrate on the fractional conversion of the frequency $\\omega \\to (...

  18. SAR processing with non-linear FM chirp waveforms.

    Energy Technology Data Exchange (ETDEWEB)

    Doerry, Armin Walter

    2006-12-01

    Nonlinear FM (NLFM) waveforms offer a radar matched filter output with inherently low range sidelobes. This yields a 1-2 dB advantage in Signal-to-Noise Ratio over the output of a Linear FM (LFM) waveform with equivalent sidelobe filtering. This report presents details of processing NLFM waveforms in both range and Doppler dimensions, with special emphasis on compensating intra-pulse Doppler, often cited as a weakness of NLFM waveforms.

  19. Model check of continue time Markov reward process with impulse reward%带动作回报的连续时间Markov回报过程验证

    Institute of Scientific and Technical Information of China (English)

    黄镇谨; 陆阳; 杨娟; 王智文

    2015-01-01

    为了能够更准确的表达不确定性复杂系统的时空验证,针对当前连续时间Markov回报过程( continue time markov reward decision process,CMRDP)验证中只考虑状态回报的问题,提出带动作回报的验证方法. 考虑添加了动作回报的空间性能约束,扩展现有的基于状态回报的连续时间Markov回报过程,用正则表达式表示验证属性的路径规范,扩展已有路径算子的表达能力. 给出带动作回报 CMRDP和路径规范的积模型,求解积模型在确定性策略下的诱导Markov回报模型( markov reward model,MRM) ,将CMRDP上的时空性能验证转换为MRM模型上的时空可达概率分析,并提出MRM中求解可达概率的算法. 实例分析表明,提出的验证思路和验证算法是可行的.%In order to express the verification of the uncertainty temporal and spatial properties of the complex sys-tems more accurately which include nondeterministic choices, continuous time Markov reward decision process based on state reward is extended by considering spatial properties with impulse reward and is adopted as verifica-tion model.The path formulas expressed by traditional path operator is replaced by regular expressions, which can express more comprehensive verification properties.Under deterministic schedulers, the induced Markov reward model of product model which is the product of continuous time Markov reward decision process with impulse reward and path formula is proposed.After that we reduce the problem of model checking for product model to the problem of computing the maximum time-and space-bound reachability probabilities of induced MRM and put forward a algo-rithm to solve this problem.The experiment results show that the model checking approach for CMRDP with im-pulse reward is feasible.

  20. A comparison of nonlinear media for parametric all-optical signal processing

    DEFF Research Database (Denmark)

    Martinez Diaz, Jordi; Bohigas Nadal, Jaume; Vukovic, Dragana;

    2013-01-01

    We systematically compare nonlinear media for parametric signal processing by determining the minimum pump power that is required for a given conversion efficiency in a degenerate four-wave mixing process, including the effect of nonlinear loss....

  1. Nonlinear Statistical Signal Processing: A Particle Filtering Approach

    Energy Technology Data Exchange (ETDEWEB)

    Candy, J

    2007-09-19

    A introduction to particle filtering is discussed starting with an overview of Bayesian inference from batch to sequential processors. Once the evolving Bayesian paradigm is established, simulation-based methods using sampling theory and Monte Carlo realizations are discussed. Here the usual limitations of nonlinear approximations and non-gaussian processes prevalent in classical nonlinear processing algorithms (e.g. Kalman filters) are no longer a restriction to perform Bayesian inference. It is shown how the underlying hidden or state variables are easily assimilated into this Bayesian construct. Importance sampling methods are then discussed and shown how they can be extended to sequential solutions implemented using Markovian state-space models as a natural evolution. With this in mind, the idea of a particle filter, which is a discrete representation of a probability distribution, is developed and shown how it can be implemented using sequential importance sampling/resampling methods. Finally, an application is briefly discussed comparing the performance of the particle filter designs with classical nonlinear filter implementations.

  2. Predicting speech intelligibility in conditions with nonlinearly processed noisy speech

    DEFF Research Database (Denmark)

    Jørgensen, Søren; Dau, Torsten

    2013-01-01

    The speech-based envelope power spectrum model (sEPSM; [1]) was proposed in order to overcome the limitations of the classical speech transmission index (STI) and speech intelligibility index (SII). The sEPSM applies the signal-tonoise ratio in the envelope domain (SNRenv), which was demonstrated...... to successfully predict speech intelligibility in conditions with nonlinearly processed noisy speech, such as processing with spectral subtraction. Moreover, a multiresolution version (mr-sEPSM) was demonstrated to account for speech intelligibility in various conditions with stationary and fluctuating...... from computational auditory scene analysis and further support the hypothesis that the SNRenv is a powerful metric for speech intelligibility prediction....

  3. Recursive utility in a Markov environment with stochastic growth.

    Science.gov (United States)

    Hansen, Lars Peter; Scheinkman, José A

    2012-07-24

    Recursive utility models that feature investor concerns about the intertemporal composition of risk are used extensively in applied research in macroeconomics and asset pricing. These models represent preferences as the solution to a nonlinear forward-looking difference equation with a terminal condition. In this paper we study infinite-horizon specifications of this difference equation in the context of a Markov environment. We establish a connection between the solution to this equation and to an arguably simpler Perron-Frobenius eigenvalue equation of the type that occurs in the study of large deviations for Markov processes. By exploiting this connection, we establish existence and uniqueness results. Moreover, we explore a substantive link between large deviation bounds for tail events for stochastic consumption growth and preferences induced by recursive utility.

  4. Bayesian framework for modeling diffusion processes with nonlinear drift based on nonlinear and incomplete observations.

    Science.gov (United States)

    Wu, Hao; Noé, Frank

    2011-03-01

    Diffusion processes are relevant for a variety of phenomena in the natural sciences, including diffusion of cells or biomolecules within cells, diffusion of molecules on a membrane or surface, and diffusion of a molecular conformation within a complex energy landscape. Many experimental tools exist now to track such diffusive motions in single cells or molecules, including high-resolution light microscopy, optical tweezers, fluorescence quenching, and Förster resonance energy transfer (FRET). Experimental observations are most often indirect and incomplete: (1) They do not directly reveal the potential or diffusion constants that govern the diffusion process, (2) they have limited time and space resolution, and (3) the highest-resolution experiments do not track the motion directly but rather probe it stochastically by recording single events, such as photons, whose properties depend on the state of the system under investigation. Here, we propose a general Bayesian framework to model diffusion processes with nonlinear drift based on incomplete observations as generated by various types of experiments. A maximum penalized likelihood estimator is given as well as a Gibbs sampling method that allows to estimate the trajectories that have caused the measurement, the nonlinear drift or potential function and the noise or diffusion matrices, as well as uncertainty estimates of these properties. The approach is illustrated on numerical simulations of FRET experiments where it is shown that trajectories, potentials, and diffusion constants can be efficiently and reliably estimated even in cases with little statistics or nonequilibrium measurement conditions.

  5. Nonlinear processes in the strong wave-plasma interaction

    Science.gov (United States)

    Pegoraro, Francesco; Califano, Francesco; Attico, Nicola; Bulanov, Sergei

    2000-10-01

    Nonlinear interactions in hot laboratory and/or astrophysical plasmas are a very efficient mechanism able to transfer the energy from the large to the small spatial scales of the system. As a result, kinetic processes are excited and play a key role in the plasma dynamics since the typical fluid dissipative length scales (where the nonlinear cascade is stopped) are (much) smaller then the kinetic length scales. Then, the key point is the role of the kinetic effects in the global plasma dynamics, i.e. whether the kinetic effects remains confined to the small scales of the system or whether there is a significant feedback on the large scales. Here we will address this problem by discussing the nonlinear kinetic evolution of the electromagnetic beam plasma instability where phase space vortices, as well as large scale vortex like magnetic structures in the physical space, are generated by wave - particle interactions. The role and influence of kinetic effects on the large scale plasma dynamics will be also discussed by addressing the problem of collisionless magnetic reconection.

  6. Experimental characterization of nonlinear processes of whistler branch waves

    Science.gov (United States)

    Tejero, E. M.; Crabtree, C.; Blackwell, D. D.; Amatucci, W. E.; Ganguli, G.; Rudakov, L.

    2016-05-01

    Experiments in the Space Physics Simulation Chamber at the Naval Research Laboratory isolated and characterized important nonlinear wave-wave and wave-particle interactions that can occur in the Earth's Van Allen radiation belts by launching predominantly electrostatic waves in the intermediate frequency range with wave normal angle greater than 85 ° and measuring the nonlinearly generated electromagnetic scattered waves. The scattered waves have a perpendicular wavelength that is nearly an order of magnitude larger than that of the pump wave. Calculations of scattering efficiency from experimental measurements demonstrate that the scattering efficiency is inversely proportional to the damping rate and trends towards unity as the damping rate approaches zero. Signatures of both wave-wave and wave-particle scatterings are also observed in the triggered emission process in which a launched wave resonant with a counter-propagating electron beam generates a large amplitude chirped whistler wave. The possibility of nonlinear scattering or three wave decay as a saturation mechanism for the triggered emission is suggested. The laboratory experiment has inspired the search for scattering signatures in the in situ data of chorus emission in the radiation belts.

  7. Recent Advances in Graphene-Assisted Nonlinear Optical Signal Processing

    Directory of Open Access Journals (Sweden)

    Jian Wang

    2016-01-01

    Full Text Available Possessing a variety of remarkable optical, electronic, and mechanical properties, graphene has emerged as an attractive material for a myriad of optoelectronic applications. The wonderful optical properties of graphene afford multiple functions of graphene based polarizers, modulators, transistors, and photodetectors. So far, the main focus has been on graphene based photonics and optoelectronics devices. Due to the linear band structure allowing interband optical transitions at all photon energies, graphene has remarkably large third-order optical susceptibility χ(3, which is only weakly dependent on the wavelength in the near-infrared frequency range. The graphene-assisted four-wave mixing (FWM based wavelength conversions have been experimentally demonstrated. So, we believe that the potential applications of graphene also lie in nonlinear optical signal processing, where the combination of its unique large χ(3 nonlinearities and dispersionless over the wavelength can be fully exploited. In this review article, we give a brief overview of our recent progress in graphene-assisted nonlinear optical device and their applications, including degenerate FWM based wavelength conversion of quadrature phase-shift keying (QPSK signal, phase conjugated wavelength conversion by degenerate FWM and transparent wavelength conversion by nondegenerate FWM, two-input and three-input high-base optical computing, and high-speed gate-tunable terahertz coherent perfect absorption (CPA using a split-ring graphene.

  8. Quantum Markov fields on graphs

    OpenAIRE

    2009-01-01

    We introduce generalized quantum Markov states and generalized d-Markov chains which extend the notion quantum Markov chains on spin systems to that on $C^*$-algebras defined by general graphs. As examples of generalized d-Markov chains, we construct the entangled Markov fields on tree graphs. The concrete examples of generalized d-Markov chains on Cayley trees are also investigated.

  9. Generation of large fifth-order nonlinear phase shifts by use of lossless χ(2) cascaded nonlinear processes

    Institute of Scientific and Technical Information of China (English)

    XU Guang; QIAN Liejia; WANG Tao; FAN Dianyuan; LI Fuming

    2004-01-01

    It is shown that the cascaded fifth-order nonlinear phase shifts will increase with energy loss in the cascaded processes. Essentially different from the multi-photon absorption accompanied with inherent material nonlinearities, the loss of fundamental wave in a cascaded process is controllable and suppressible. By introducing difference frequencies generated from the reaction between the fundamental and its second harmonic after the cascaded processes, the fundamental wave can be free of energy loss, while the large cascaded fifth-order nonlinear phase shift is maintained.

  10. Markov决策过程在矿井通讯系统中的应用%Application of Markov Decision Process in Coal Mine Communication System

    Institute of Scientific and Technical Information of China (English)

    沈晋会

    2013-01-01

    With the rapid development of network technology, information system processing power is more and more strong. In the mine communication system application Markov decision process automation management gradually to the direction of expansion, and can be widely used in coal mine communication management, realize information management for our country modernization has a very important significance. Based on the analysis of the advantages and disadvantages of Markov decision process, based on the proposed in coal mine communication system application and reliable transmission technology, design management system of coal mine Markov decision model, and in this mechanism into the transport layer information feedback, in order to achieve coal mine communication system information application of optimization.%随着网络技术的快速发展,信息化系统的处理能力越来越强。在矿井通讯系统中应用Markov决策过程逐渐向自动化管理方向扩展,并得到更广泛的应用,在煤矿通讯管理中实现信息化管理对于我国现代化建设有着非常重大的意义。文章在分析Markov决策过程的优缺点基础上,提出在煤矿通讯系统中应用可靠传输技术,设计煤矿管理系统Markov决策模型,并在该机制中引入传输层信息的反馈,以达到煤矿通讯系统信息化应用的最优化。

  11. The Estimation of the Semi-Markov Processes Performance Potentials Based on Parallel Simulation%半Markov过程性能势的并行仿真估计

    Institute of Scientific and Technical Information of China (English)

    周亚平; 刘剑宇; 殷保群; 奚宏生

    2005-01-01

    给出半Markov过程(Semi-Markov Processes)性能势基于一条样本轨道的仿真算法,从并行仿真的角度,将已有Markov过程的性能势理论推广到半Markov过程,使该理论具有更加广泛的应用范围.并将该性能势与等价的Markov过程的性能势进行比较,表明了两者的一致性.

  12. Nonlinear Optical Microscopy Signal Processing Strategies in Cancer

    Science.gov (United States)

    Adur, Javier; Carvalho, Hernandes F; Cesar, Carlos L; Casco, Víctor H

    2014-01-01

    This work reviews the most relevant present-day processing methods used to improve the accuracy of multimodal nonlinear images in the detection of epithelial cancer and the supporting stroma. Special emphasis has been placed on methods of non linear optical (NLO) microscopy image processing such as: second harmonic to autofluorescence ageing index of dermis (SAAID), tumor-associated collagen signatures (TACS), fast Fourier transform (FFT) analysis, and gray level co-occurrence matrix (GLCM)-based methods. These strategies are presented as a set of potential valuable diagnostic tools for early cancer detection. It may be proposed that the combination of NLO microscopy and informatics based image analysis approaches described in this review (all carried out on free software) may represent a powerful tool to investigate collagen organization and remodeling of extracellular matrix in carcinogenesis processes. PMID:24737930

  13. Markov stochasticity coordinates

    Science.gov (United States)

    Eliazar, Iddo

    2017-01-01

    Markov dynamics constitute one of the most fundamental models of random motion between the states of a system of interest. Markov dynamics have diverse applications in many fields of science and engineering, and are particularly applicable in the context of random motion in networks. In this paper we present a two-dimensional gauging method of the randomness of Markov dynamics. The method-termed Markov Stochasticity Coordinates-is established, discussed, and exemplified. Also, the method is tweaked to quantify the stochasticity of the first-passage-times of Markov dynamics, and the socioeconomic equality and mobility in human societies.

  14. Markov stochasticity coordinates

    Energy Technology Data Exchange (ETDEWEB)

    Eliazar, Iddo, E-mail: iddo.eliazar@intel.com

    2017-01-15

    Markov dynamics constitute one of the most fundamental models of random motion between the states of a system of interest. Markov dynamics have diverse applications in many fields of science and engineering, and are particularly applicable in the context of random motion in networks. In this paper we present a two-dimensional gauging method of the randomness of Markov dynamics. The method–termed Markov Stochasticity Coordinates–is established, discussed, and exemplified. Also, the method is tweaked to quantify the stochasticity of the first-passage-times of Markov dynamics, and the socioeconomic equality and mobility in human societies.

  15. Determinação da capacidade real necessária de um processo produtivo utilizando cadeia de Markov Determination of necessary real capacity in productive process using Markov chain

    Directory of Open Access Journals (Sweden)

    Francielly Hedler Staudt

    2011-01-01

    Full Text Available Todas as empresas em desenvolvimento passam pelo momento de decidir se há ou não necessidade de realizar novos investimentos para suprir uma demanda crescente. Para tomar tal decisão é imprescindível conhecer se o processo atual tem capacidade de produzir a nova demanda. Porém, são raras as empresas que têm a percepção de que os refugos e retrabalhos também consomem recursos da produção e, portanto, devem ser considerados no cálculo da capacidade produtiva. A proposta deste trabalho consiste em incluir esses fatores na análise de capacidade da fábrica, utilizando uma matriz de transição estocástica da cadeia absorvente de Markov como ferramenta para obtenção do fator de capacidade. Este fator, aliado ao índice de eficiência e a demanda desejada ao fim do processo, resulta na capacidade real necessária. Um estudo de caso exemplifica a metodologia, apresentando resultados que permitem o cálculo do índice de ocupação real de cada centro produtivo. O cálculo desse índice demonstrou que alguns centros de trabalho necessitam de análises sobre investimentos em capacitação, pois ultrapassaram 90% de ocupação.All developing companies must decide once in a while whether it is required to perform new investments to handle a growing demand. In order to make this decision, it is essential to know whether the current productive capacity is able to supply the new demand. However, just few companies realize that refuse and rework use production resources, which must be taken into account in the productive capacity calculation. The aim of this work was to include these factors in factory capacity analysis, using Markov chain stochastic transition matrix as a tool to obtain the capacity factor. This factor - used together with the efficiency index and the required demand in the end of the process - results in the necessary real capacity. A case study exemplifies the proposed methodology, presenting results that allow for the

  16. 基于 MarKov 过程的导航系统星座可用性分析%Availability analysis for constellation of GNSS based on MarKov process

    Institute of Scientific and Technical Information of China (English)

    侯洪涛; 谢菲; 张旺勋; 王维平; 黄丛山

    2014-01-01

    It is a contradiction between accuracy and rapidity of availability analysis for constellation of the global navigation satellite system (GNSS).The requirements for availability analysis are researched firstly,and then the basic definition for availability of the constellation is analyzed,the method for analysis based on failure subsystem is proposed,the algorithm for single satellite based on Markov process is given,and the model of service availability based on constellation state probability is developed,the availability of the constellation can be calculated with the model and the basic principles of navigation.Finally,the regional availability performance of the BeiDou satellite navigation system is analyzed based on the proposed architecture and method.The results show that the proposed methods and models meet the special needs for availability analysis of GNSS.%卫星导航系统星座可用性分析面临准确性和快速性的矛盾。论文在研究全球卫星导航系统(global navigation satellite system,GNSS)星座可用性分析需求的基础上,系统地定义了导航系统可用性的基本概念,给出了基于分系统部件失效计算导航星座可用性的方法,提出了基于 Markov 过程的单星可用度算法,建立了基于星座状态概率的服务可用性计算模型,并结合导航基本原理模型得到了导航星座的可用性。最后,基于此方法针对北斗区域卫星导航系统相关数据对其星座可用性进行了仿真和实验分析。结果表明,本文所提方法和模型能够满足卫星导航系统星座可用性分析的特殊需求。

  17. A simple nonlinear PD controller for integrating processes.

    Science.gov (United States)

    Dey, Chanchal; Mudi, Rajani K; Simhachalam, Dharmana

    2014-01-01

    Many industrial processes are found to be integrating in nature, for which widely used Ziegler-Nichols tuned PID controllers usually fail to provide satisfactory performance due to excessive overshoot with large settling time. Although, IMC (Internal Model Control) based PID controllers are capable to reduce the overshoot, but little improvement is found in the load disturbance response. Here, we propose an auto-tuning proportional-derivative controller (APD) where a nonlinear gain updating factor α continuously adjusts the proportional and derivative gains to achieve an overall improved performance during set point change as well as load disturbance. The value of α is obtained by a simple relation based on the instantaneous values of normalized error (eN) and change of error (ΔeN) of the controlled variable. Performance of the proposed nonlinear PD controller (APD) is tested and compared with other PD and PID tuning rules for pure integrating plus delay (IPD) and first-order integrating plus delay (FOIPD) processes. Effectiveness of the proposed scheme is verified on a laboratory scale servo position control system.

  18. Quantum Markov Chain Mixing and Dissipative Engineering

    DEFF Research Database (Denmark)

    Kastoryano, Michael James

    2012-01-01

    This thesis is the fruit of investigations on the extension of ideas of Markov chain mixing to the quantum setting, and its application to problems of dissipative engineering. A Markov chain describes a statistical process where the probability of future events depends only on the state...... of the system at the present point in time, but not on the history of events. Very many important processes in nature are of this type, therefore a good understanding of their behaviour has turned out to be very fruitful for science. Markov chains always have a non-empty set of limiting distributions...... (stationary states). The aim of Markov chain mixing is to obtain (upper and/or lower) bounds on the number of steps it takes for the Markov chain to reach a stationary state. The natural quantum extensions of these notions are density matrices and quantum channels. We set out to develop a general mathematical...

  19. Confluence reduction for Markov automata

    NARCIS (Netherlands)

    Timmer, Mark; Pol, van de Jaco; Stoelinga, Mariëlle

    2013-01-01

    Markov automata are a novel formalism for specifying systems exhibiting nondeterminism, probabilistic choices and Markovian rates. Recently, the process algebra MAPA was introduced to efficiently model such systems. As always, the state space explosion threatens the analysability of the models gener

  20. Hierarchical Markov decision processes based path planning for UAV in three-dimensional environment%基于HMDP的无人机三维路径规划

    Institute of Scientific and Technical Information of China (English)

    洪晔; 房建成

    2009-01-01

    路径规划是UAV(Unmanned Aerial Vehicle)自主飞行的重要保障.初步建立了基于MDP(Markov Decision Processes)的全局路径规划模型,把UAV的路径规划看作是给定环境模型和奖惩原则的情况下,寻求最优策略的问题;为解决算法时空开销大、UAV航向改变频繁的缺点,提出一种基于状态聚类方法的HMDP(Hierarchical Markov Decision Processes)模型,并将其拓展到三维规划中.仿真实验证明:这种简单的规划模型可以有效解决UAV的三维全局路径规划问题,为其在实际飞行中的局部规划奠定了基础.

  1. POISSON LIMIT THEOREM FOR COUNTABLE MARKOV CHAINS IN MARKOVIAN ENVIRONMENTS

    Institute of Scientific and Technical Information of China (English)

    方大凡; 王汉兴; 唐矛宁

    2003-01-01

    A countable Markov chain in a Markovian environment is considered. A Poisson limit theorem for the chain recurring to small cylindrical sets is mainly achieved. In order to prove this theorem, the entropy function h is introduced and the Shannon-McMillan-Breiman theorem for the Markov chain in a Markovian environment is shown. It' s well-known that a Markov process in a Markovian environment is generally not a standard Markov chain, so an example of Poisson approximation for a process which is not a Markov process is given. On the other hand, when the environmental process degenerates to a constant sequence, a Poisson limit theorem for countable Markov chains, which is the generalization of Pitskel's result for finite Markov chains is obtained.

  2. Nonlinear calibration and data processing of the solar radio burst

    Institute of Scientific and Technical Information of China (English)

    颜毅华; 谭程明; 徐龙; 姬慧荣; 傅其骏; 宋国乡

    2002-01-01

    The processes of the sudden energy release and energy transfer, and particle accelerations are the most challenge fundamental problems in solar physics as well as in astrophysics. Nowadays, there has been no direct measurement of the plasma parameters and magnetic fields at the coronal energy release site. Under the certain hypothesis of radiation mechanism and transmission process, radio measurement is almost the only method to diagnose coronal magnetic field. The broadband dynamic solar radio spectrometer that has been finished recently in China has higher time and frequency resolutions. Thus it plays an important role during the research of the 23rd solar cycle in China. Sometimes when there were very large bursts, the spectrometer will be overflowed. It needs to take some special process to discriminate the instrument and interference effects from solar burst signals. According to the characteristic of the solar radio broadband dynamic spectrometer, we developed a nonlinear calibration method to deal with the overflow of instrument, and introduced channel-modification method to deal with images. Finally the interference is eliminated with the help of the wavelet method. Here we take the analysis of the well-known solar-terrestrial event on July 14th, 2000 as the example. It shows the feasibility and validity of the method mentioned above. These methods can also be applied to other issues.

  3. Nonlinear closure relations theory for transport processes in nonequilibrium systems.

    Science.gov (United States)

    Sonnino, Giorgio

    2009-05-01

    A decade ago, a macroscopic theory for closure relations has been proposed for systems out of Onsager's region. This theory is referred to as the thermodynamic field theory (TFT). The aim of this work was to determine the nonlinear flux-force relations that respect the thermodynamic theorems for systems far from equilibrium. We propose a formulation of the TFT where one of the basic restrictions, namely, the closed-form solution for the skew-symmetric piece of the transport coefficients, has been removed. In addition, the general covariance principle is replaced by the De Donder-Prigogine thermodynamic covariance principle (TCP). The introduction of TCP requires the application of an appropriate mathematical formalism, which is referred to as the entropy-covariant formalism. By geometrical arguments, we prove the validity of the Glansdorff-Prigogine universal criterion of evolution. A new set of closure equations determining the nonlinear corrections to the linear ("Onsager") transport coefficients is also derived. The geometry of the thermodynamic space is non-Riemannian. However, it tends to be Riemannian for high values of the entropy production. In this limit, we recover the transport equations found by the old theory. Applications of our approach to transport in magnetically confined plasmas, materials submitted to temperature, and electric potential gradients or to unimolecular triangular chemical reactions can be found at references cited herein. Transport processes in tokamak plasmas are of particular interest. In this case, even in the absence of turbulence, the state of the plasma remains close to (but, it is not in) a state of local equilibrium. This prevents the transport relations from being linear.

  4. Nonlinear properties of and nonlinear processing in hydrogenated amorphous silicon waveguides

    DEFF Research Database (Denmark)

    Kuyken, B.; Ji, Hua; Clemmen, S.

    2011-01-01

    We propose hydrogenated amorphous silicon nanowires as a platform for nonlinear optics in the telecommunication wavelength range. Extraction of the nonlinear parameter of these photonic nanowires reveals a figure of merit larger than 2. It is observed that the nonlinear optical properties...... of these waveguides degrade with time, but that this degradation can be reversed by annealing the samples. A four wave mixing conversion efficiency of + 12 dB is demonstrated in a 320 Gbit/s serial optical waveform data sampling experiment in a 4 mm long photonic nanowire....

  5. Nonlinear quantum electrodynamic and electroweak processes in strong laser fields

    Energy Technology Data Exchange (ETDEWEB)

    Meuren, Sebastian

    2015-06-24

    Various nonlinear electrodynamic and electroweak processes in strong plane-wave laser fields are considered with an emphasis on short-pulse effects. In particular, the momentum distribution of photoproduced electron-positron pairs is calculated numerically and a semiclassical interpretation of its characteristic features is established. By proving the optical theorem, compact double-integral expressions for the total pair-creation probability are obtained and numerically evaluated. The exponential decay of the photon wave function in a plane wave is included by solving the Schwinger-Dyson equations to leading-order in the quasistatic approximation. In this respect, the polarization operator in a plane wave is investigated and its Ward-Takahashi identity verified. A classical analysis indicates that a photoproduced electron-positron pair recollides for certain initial conditions. The contributions of such recollision processes to the polarization operator are identified and calculated both analytically and numerically. Furthermore, the existence of nontrivial electron-spin dynamics induced by quantum fluctuations is verified for ultra-short laser pulses. Finally, the exchange of weak gauge bosons is considered, which is essential for neutrino-photon interactions. In particular, the axial-vector-vector coupling tensor is calculated and the so-called Adler-Bell-Jackiw (ABJ) anomaly investigated.

  6. Fields From Markov Chains

    DEFF Research Database (Denmark)

    Justesen, Jørn

    2005-01-01

    A simple construction of two-dimensional (2-D) fields is presented. Rows and columns are outcomes of the same Markov chain. The entropy can be calculated explicitly.......A simple construction of two-dimensional (2-D) fields is presented. Rows and columns are outcomes of the same Markov chain. The entropy can be calculated explicitly....

  7. Decisive Markov Chains

    CERN Document Server

    Abdulla, Parosh Aziz; Mayr, Richard

    2007-01-01

    We consider qualitative and quantitative verification problems for infinite-state Markov chains. We call a Markov chain decisive w.r.t. a given set of target states F if it almost certainly eventually reaches either F or a state from which F can no longer be reached. While all finite Markov chains are trivially decisive (for every set F), this also holds for many classes of infinite Markov chains. Infinite Markov chains which contain a finite attractor are decisive w.r.t. every set F. In particular, this holds for probabilistic lossy channel systems (PLCS). Furthermore, all globally coarse Markov chains are decisive. This class includes probabilistic vector addition systems (PVASS) and probabilistic noisy Turing machines (PNTM). We consider both safety and liveness problems for decisive Markov chains, i.e., the probabilities that a given set of states F is eventually reached or reached infinitely often, respectively. 1. We express the qualitative problems in abstract terms for decisive Markov chains, and show...

  8. All-optical signal processing in quadratic nonlinear materials

    DEFF Research Database (Denmark)

    Johansen, Steffen Kjær

    2002-01-01

    of materials with a second order nonlinearity, the so-called X(2) materials, is faster and stronger than that of more conventional materials with a cubic nonlinearity. The X(2) materials support spatial solitons consisting of two coupled components, the fundamental wave (FW) and its second harmonic (SH......). During this project the interaction between such spatial solitons has been investigated theoretically through perturbation theory and experimentally via numerical simulations. The outcome of this research isnew theoretical tools for quantitatively predicting the escape angle, i.e. the angle of incidence...... and exploitation of these cubic nonlinearities in two-period QPM wave-guides has been another area of investigation. Introducing the second period might make practical engineering of the nonlinearities possible. A major result is the discovery that cubic nonlinearities leads to an enhancement of the bandwidth...

  9. Average Optimality in Markov Decision Processes with Unbounded Rewards%报酬无界的平均准则马氏决策过程

    Institute of Scientific and Technical Information of China (English)

    胡奇英

    2002-01-01

    本文对可数状态集、非空决策集、报酬无界的平均准则马氏决策过程,提出了一组新的条件,在此条件下存在(ε)最优平稳策略,且当最优不等式中的和有定义时最优不等式也成立.%This paper studies average optimality in Markov decision processes with countablestate space, nonempty action sets and unbounded reward function. New conditions arediscussed under which there exists an (ε) optimal stationary policy, and that the averagecriterion optimality inequality holds when the summation in it is well defined.

  10. 一种部分可感知系统的增强学习方法%A Reinforcement Learning Algorithm for Partially Observable Markov Decision Processes

    Institute of Scientific and Technical Information of China (English)

    汤俏; 赵凯

    2004-01-01

    @@ 1引言 在人工智能领域中,增强学习理论由于其自学习性和自适应性的优点而得到了广泛关注,在机器人控制系统,优化组合问题等诸多领域得到了越来越广泛的应用,是当前研究的重点问题之一[1].现有的增强学习方法对马尔可夫决策过程(MDP,Markov Decision Processes),即,进行策略选择的agent能够准确全面地获得关于环境所有信息的情况,已经有了多种较成熟的算法,如Q-learning等[2,3].

  11. Nonuniform Markov models

    CERN Document Server

    Ristad, E S; Ristad, Eric Sven; Thomas, Robert G.

    1996-01-01

    A statistical language model assigns probability to strings of arbitrary length. Unfortunately, it is not possible to gather reliable statistics on strings of arbitrary length from a finite corpus. Therefore, a statistical language model must decide that each symbol in a string depends on at most a small, finite number of other symbols in the string. In this report we propose a new way to model conditional independence in Markov models. The central feature of our nonuniform Markov model is that it makes predictions of varying lengths using contexts of varying lengths. Experiments on the Wall Street Journal reveal that the nonuniform model performs slightly better than the classic interpolated Markov model. This result is somewhat remarkable because both models contain identical numbers of parameters whose values are estimated in a similar manner. The only difference between the two models is how they combine the statistics of longer and shorter strings. Keywords: nonuniform Markov model, interpolated Markov m...

  12. Minimal Markov Models

    CERN Document Server

    Gonzalez-Lopez, Jesus E Garcia Veronica A

    2010-01-01

    In this work we introduce a new and richer class of finite order Markov chain models and address the following model selection problem: find the Markov model with the minimal set of parameters (minimal Markov model) which is necessary to represent a source as a Markov chain of finite order. Let us call $M$ the order of the chain and $A$ the finite alphabet, to determine the minimal Markov model, we define an equivalence relation on the state space $A^{M}$, such that all the sequences of size $M$ with the same transition probabilities are put in the same category. In this way we have one set of $(|A|-1)$ transition probabilities for each category, obtaining a model with a minimal number of parameters. We show that the model can be selected consistently using the Bayesian information criterion.

  13. Markov决策过程不确定策略特征模式%Property Patterns of Markov Decision Process Nondeterministic Choice Scheduler

    Institute of Scientific and Technical Information of China (English)

    黄镇谨; 陆阳; 杨娟; 方欢

    2013-01-01

    Markov decision process can model complex system with nondeterminism. Schedulers are required to resolve the nonderministic choices during model analysis. This paper introduced the time-and space-bounded reachability probabilities of markov decision process under different schedulers. Firstly,the formal definition and classification method of schedulers for nondertninism were proposed and then we proved that the reachability probabilities coincide for deterministic and randomized schedulers under time-abstract Also, it was proved that time-dependent scheduler generally induces probability bounds that exceed those of the corresponding time-abstract At the end of paper, two cases were illustrated for describing the correctness of the conclusion.%马尔科夫决策过程可以建模具有不确定性特征的复杂系统,而在进行模型分析时需要采用策略对不确定性进行处理.首先,研究不同策略下时空有界可达概率问题,给出不确定性解决策略的定义及分类方法.其次,在时间无关策略下,证明基于确定性选取动作和随机选取动作的时空有界可达概率的一致性,并且论证了时间依赖策略相对于时间无关策略具有更好的时空有界可达概率.最后结合实例简要阐述了结论的正确性.

  14. Maintenance scheduling for semiconductor manufacturing equipment based on Markov Decision Process%基于MDP的半导体制造设备维护调度研究

    Institute of Scientific and Technical Information of China (English)

    陈静静

    2012-01-01

    To solve the maintenance scheduling problem of semiconductor manufacturing equipment, two-layer maintenance optimization model based on Markov decision process is proposed. Device layer uses Markov Decision Process to derive the optimal policy for job sequencing,cleaning and maintenance of each device in the long-term, both considering the degradation failure and stochastic failure. Genetic algorithm is applied to assign a limited number of maintenance personnels to manufacturing equipment in order to induce the productivity losses. Finally this model based on a certain semiconductor wafer fabrication is built on Em-Plant, and results revealed satisfactory scheduling performance.%为解决半导体制造设备的维护调度问题,提出基于马尔可夫决策过程(MDP)的制造设备两层维护优化模型.设备层利用马尔可夫决策过程(MDP)模型,同时考虑劣化故障和随机故障两种故障类型,制定针对单台设备的工件排序、清洗和维修的长期维护优化策略,系统层采用遗传算法解决有限维护资源下短期维修调度方案,尽可能降低由设备故障所导致的生产损失.最后以某半导体生产线为例利用eM-Plant软件进行仿真验证,结果表明,该维护策略能更好的提高系统性能.

  15. New Algorithm Model for Processing Generalized Dynamic Nonlinear Data Derived from Deformation Monitoring Network

    Institute of Scientific and Technical Information of China (English)

    LIN Xiangguo; LIANG Yong

    2005-01-01

    The processing of nonlinear data was one of hot topics in surveying and mapping field in recent years.As a result, many linear methods and nonlinear methods have been developed.But the methods for processing generalized nonlinear surveying and mapping data, especially for different data types and including unknown parameters with random or nonrandom, are seldom noticed.A new algorithm model is presented in this paper for processing nonlinear dynamic multiple-period and multiple-accuracy data derived from deformation monitoring network.

  16. Quantitative Safety Analysis of Train Control System Based on Markov Decision Process%基于 Markov 决策过程的列控系统定量安全分析方法

    Institute of Scientific and Technical Information of China (English)

    周果; 赵会兵

    2016-01-01

    In order to manage the hazards in the phase of train control system design and safety assessment , the quantitative safety analysis of train control system is crucial . T he results of the quantitative analysis can be used to judge and compare the pros and cons of the prototype designs , to evaluate the probabilistic risks of haz‐ards and to determine w hether hidden dangers can be controlled within the acceptable range with the risk miti‐gation measures taken . In this paper , a Markov Decision Process based modelling method was proposed to build system behavior model of the two consecutive trains in the train control system , to integrate the normal behaviours and failure behaviours of the system and to put forward Comprehensive Behaviour Model (CBM ) . The dangerous failure probability of the hazard was calculated under the probabilistic model checking tool PRISM . The methodology of quantitative safety analysis for train control system was presented .%为了在列控系统的设计阶段和安全评估阶段对系统隐患进行把握,对系统的设计进行定量安全分析是至关重要的。定量分析的结果可以用来判断和比较设计的优劣,也可用来评估隐患的风险,并根据分析结果判断所采取的隐患控制措施是否使隐患的风险被控制在可接受的范围内。本文应用以Markov决策过程为基础的建模方法,对列控系统中的双车追踪场景进行系统行为建模,集成系统正常行为和失效行为,提出综合系统行为模型CBM ,并通过概率模型检验工具 PRISM对危险失效概率进行准确计算,提出列控系统定量安全分析方法。

  17. Application of Novel Nonlinear Optical Materials to Optical Processing

    Science.gov (United States)

    Banerjee, Partha P.

    1999-01-01

    We describe wave mixing and interactions in nonlinear photorefractive polymers and disodium flourescein. Higher diffracted orders yielding forward phase conjugation can be generated in a two-wave mixing geometry in photorefractive polymers, and this higher order can be used for image edge enhancement and correlation. Four-wave mixing and phase conjugation is studied using nonlinear disodium floureschein, and the nature and properties of gratings written in this material are investigated.

  18. Fuzzy Markov chains: uncertain probabilities

    OpenAIRE

    2002-01-01

    We consider finite Markov chains where there are uncertainties in some of the transition probabilities. These uncertainties are modeled by fuzzy numbers. Using a restricted fuzzy matrix multiplication we investigate the properties of regular, and absorbing, fuzzy Markov chains and show that the basic properties of these classical Markov chains generalize to fuzzy Markov chains.

  19. The Limit Behaviour of Imprecise Continuous-Time Markov Chains

    Science.gov (United States)

    De Bock, Jasper

    2016-08-01

    We study the limit behaviour of a nonlinear differential equation whose solution is a superadditive generalisation of a stochastic matrix, prove convergence, and provide necessary and sufficient conditions for ergodicity. In the linear case, the solution of our differential equation is equal to the matrix exponential of an intensity matrix and can then be interpreted as the transition operator of a homogeneous continuous-time Markov chain. Similarly, in the generalised nonlinear case that we consider, the solution can be interpreted as the lower transition operator of a specific set of non-homogeneous continuous-time Markov chains, called an imprecise continuous-time Markov chain. In this context, our convergence result shows that for a fixed initial state, an imprecise continuous-time Markov chain always converges to a limiting distribution, and our ergodicity result provides a necessary and sufficient condition for this limiting distribution to be independent of the initial state.

  20. Observation uncertainty in reversible Markov chains.

    Science.gov (United States)

    Metzner, Philipp; Weber, Marcus; Schütte, Christof

    2010-09-01

    In many applications one is interested in finding a simplified model which captures the essential dynamical behavior of a real life process. If the essential dynamics can be assumed to be (approximately) memoryless then a reasonable choice for a model is a Markov model whose parameters are estimated by means of Bayesian inference from an observed time series. We propose an efficient Monte Carlo Markov chain framework to assess the uncertainty of the Markov model and related observables. The derived Gibbs sampler allows for sampling distributions of transition matrices subject to reversibility and/or sparsity constraints. The performance of the suggested sampling scheme is demonstrated and discussed for a variety of model examples. The uncertainty analysis of functions of the Markov model under investigation is discussed in application to the identification of conformations of the trialanine molecule via Robust Perron Cluster Analysis (PCCA+) .

  1. Nonlinear Model Algorithmic Control of a pH Neutralization Process

    Institute of Scientific and Technical Information of China (English)

    ZOU Zhiyun; YU Meng; WANG Zhizhen; LIU Xinghong; GUO Yuqing; ZHANG Fengbo; GUO Ning

    2013-01-01

    Control of pH neutralization processes is challenging in the chemical process industry because of their inherent strong nonlinearity.In this paper,the model algorithmic control (MAC) strategy is extended to nonlinear processes using Hammerstein model that consists of a static nonlinear polynomial function followed in series by a linear impulse response dynamic element.A new nonlinear Hammerstein MAC algorithm (named NLH-MAC) is presented in detail.The simulation control results of a pH neutralization process show that NLH-MAC gives better control performance than linear MAC and the commonly used industrial nonlinear propotional plus integral plus derivative (PID) controller.Further simulation experiment demonstrates that NLH-MAC not only gives good control response,but also possesses good stability and robustness even with large modeling errors.

  2. MARKOV CHAIN PORTFOLIO LIQUIDITY OPTIMIZATION MODEL

    Directory of Open Access Journals (Sweden)

    Eder Oliveira Abensur

    2014-05-01

    Full Text Available The international financial crisis of September 2008 and May 2010 showed the importance of liquidity as an attribute to be considered in portfolio decisions. This study proposes an optimization model based on available public data, using Markov chain and Genetic Algorithms concepts as it considers the classic duality of risk versus return and incorporating liquidity costs. The work intends to propose a multi-criterion non-linear optimization model using liquidity based on a Markov chain. The non-linear model was tested using Genetic Algorithms with twenty five Brazilian stocks from 2007 to 2009. The results suggest that this is an innovative development methodology and useful for developing an efficient and realistic financial portfolio, as it considers many attributes such as risk, return and liquidity.

  3. Markov Models in health care

    Directory of Open Access Journals (Sweden)

    Renato Cesar Sato

    2010-09-01

    Full Text Available Markov Chains provide support for problems involving decision on uncertainties through a continuous period of time. The greater availability and access to processing power through computers allow that these models can be used more often to represent clinical structures. Markov models consider the patients in a discrete state of health, and the events represent the transition from one state to another. The possibility of modeling repetitive events and time dependence of probabilities and utilities associated permits a more accurate representation of the evaluated clinical structure. These templates can be used for economic evaluation in health care taking into account the evaluation of costs and clinical outcomes, especially for evaluation of chronic diseases. This article provides a review of the use of modeling within the clinical context and the advantages of the possibility of including time for this type of study.

  4. Markov information sources

    Science.gov (United States)

    Massey, J. L.

    1975-01-01

    A regular Markov source is defined as the output of a deterministic, but noisy, channel driven by the state sequence of a regular finite-state Markov chain. The rate of such a source is the per letter uncertainty of its digits. The well-known result that the rate of a unifilar regular Markov source is easily calculable is demonstrated, where unifilarity means that the present state of the Markov chain and the next output of the deterministic channel uniquely determine the next state. At present, there is no known method to calculate the rate of a nonunifilar source. Two tentative approaches to this unsolved problem are given, namely source identical twins and the master-slave source, which appear to shed some light on the question of rate calculation for a nonunifilar source.

  5. Markov information sources

    Science.gov (United States)

    Massey, J. L.

    1975-01-01

    A regular Markov source is defined as the output of a deterministic, but noisy, channel driven by the state sequence of a regular finite-state Markov chain. The rate of such a source is the per letter uncertainty of its digits. The well-known result that the rate of a unifilar regular Markov source is easily calculable is demonstrated, where unifilarity means that the present state of the Markov chain and the next output of the deterministic channel uniquely determine the next state. At present, there is no known method to calculate the rate of a nonunifilar source. Two tentative approaches to this unsolved problem are given, namely source identical twins and the master-slave source, which appear to shed some light on the question of rate calculation for a nonunifilar source.

  6. Femtosecond Fiber Lasers Based on Dissipative Processes for Nonlinear Microscopy

    Science.gov (United States)

    Wise, Frank W.

    2012-01-01

    Recent progress in the development of femtosecond-pulse fiber lasers with parameters appropriate for nonlinear microscopy is reviewed. Pulse-shaping in lasers with only normal-dispersion components is briefly described, and the performance of the resulting lasers is summarized. Fiber lasers based on the formation of dissipative solitons now offer performance competitive with that of solid-state lasers, but with the benefits of the fiber medium. Lasers based on self-similar pulse evolution in the gain section of a laser also offer a combination of short pulse duration and high pulse energy that will be attractive for applications in nonlinear bioimaging. PMID:23869163

  7. Mathematical models of non-linear phenomena, processes and systems: from molecular scale to planetary atmosphere

    CERN Document Server

    2013-01-01

    This book consists of twenty seven chapters, which can be divided into three large categories: articles with the focus on the mathematical treatment of non-linear problems, including the methodologies, algorithms and properties of analytical and numerical solutions to particular non-linear problems; theoretical and computational studies dedicated to the physics and chemistry of non-linear micro-and nano-scale systems, including molecular clusters, nano-particles and nano-composites; and, papers focused on non-linear processes in medico-biological systems, including mathematical models of ferments, amino acids, blood fluids and polynucleic chains.

  8. Nonlinear giant magnetoimpedance and the asymmetric circumferential magnetization process in soft magnetic wires

    Science.gov (United States)

    Gómez-Polo, C.; Duque, J. G. S.; Knobel, M.

    2004-07-01

    The magnetoimpedance effect and its nonlinear terms are analysed for a (Co0.94Fe0.06)72.5Si12.5B15 amorphous wire. In order to enhance the nonlinear contribution the sample was previously subjected to current annealing (Joule heating) to induce a circumferential anisotropy. The effect of the application of a torsional strain on the nonlinear magnetoimpedance is analysed in terms of the torsional dependence of the magnetic permeability, evaluated through experimental circumferential hysteresis loops. The results obtained clearly confirm the direct correlation between the asymmetric circumferential magnetization process and the occurrence of nonlinear second-harmonic terms in the magnetoimpedance voltage.

  9. Neural Generalized Predictive Control of a non-linear Process

    DEFF Research Database (Denmark)

    Sørensen, Paul Haase; Nørgård, Peter Magnus; Ravn, Ole

    1998-01-01

    qualities. The controller is a non-linear version of the well-known generalized predictive controller developed in linear control theory. It involves minimization of a cost function which in the present case has to be done numerically. Therefore, we develop the numerical algorithms necessary in substantial...

  10. Markov invariants, plethysms, and phylogenetics (the long version)

    CERN Document Server

    Sumner, J G; Jermiin, L S; Jarvis, P D

    2008-01-01

    We explore model based techniques of phylogenetic tree inference exercising Markov invariants. Markov invariants are group invariant polynomials and are distinct from what is known in the literature as phylogenetic invariants, although we establish a commonality in some special cases. We show that the simplest Markov invariant forms the foundation of the Log-Det distance measure. We take as our primary tool group representation theory, and show that it provides a general framework for analysing Markov processes on trees. From this algebraic perspective, the inherent symmetries of these processes become apparent, and focusing on plethysms, we are able to define Markov invariants and give existence proofs. We give an explicit technique for constructing the invariants, valid for any number of character states and taxa. For phylogenetic trees with three and four leaves, we demonstrate that the corresponding Markov invariants can be fruitfully exploited in applied phylogenetic studies.

  11. APLIKASI MARKOV RANDOM FIELD PADA MASALAH INDUSTRI

    Directory of Open Access Journals (Sweden)

    Siana Halim

    2002-01-01

    Full Text Available Markov chain in the stochastic process is widely used in the industrial problems particularly in the problem of determining the market share of products. In this paper we are going to extend the one in the random field so called the Markov Random Field and applied also in the market share problem with restriction the market is considered as a discrete lattice and Pott's models are going to be used as the potential function. Metropolis sampler is going to be used to determine the stability condition. Abstract in Bahasa Indonesia : Rantai Markov dalam proses stokastik seringkali digunakan dalam penyelesaian masalah industri khususnya dalam masalah penentuan market share. Dalam artikel ini akan dibahas perluasan Rantai Markov tesebut ke dalam sebuah random field yang disebut sebagai Markov Random Field (MRF yang juga akan diaplikasikan pada masalah market share dengan batasan daerah pemasarannya dianggap sebagai sebuah lattice diskrit dan fungsi potensial yang akan digunakan adalah Potts models. Akan digunakan Metropolis sampler untuk menentukan kondisi stabil. Kata kunci: proses stokastik, Markov Random Field, Gibbs Random Field, Potts model, Metropolis sampling.

  12. White noise theory of robust nonlinear filtering with correlated state and observation noises

    NARCIS (Netherlands)

    Bagchi, Arunabha; Karandikar, Rajeeva

    1994-01-01

    In the existing `direct¿ white noise theory of nonlinear filtering, the state process is still modelled as a Markov process satisfying an Itô stochastic differential equation, while a `finitely additive¿ white noise is used to model the observation noise. We remove this asymmetry by modelling the st

  13. White noise theory of robust nonlinear filtering with correlated state and observation noises

    NARCIS (Netherlands)

    Bagchi, Arunabha; Karandikar, Rajeeva

    1992-01-01

    In the direct white noise theory of nonlinear filtering, the state process is still modeled as a Markov process satisfying an Ito stochastic differential equation, while a finitely additive white noise is used to model the observation noise. In the present work, this asymmetry is removed by modeling

  14. Nonlinear analysis and control of a continuous fermentation process

    DEFF Research Database (Denmark)

    Szederkényi, G.; Kristensen, Niels Rode; Hangos, K.M

    2002-01-01

    open-loop system properties, to explore the possible control difficulties and to select the system output to be used in the control structure. A wide range of controllers are tested including pole placement and LQ controllers, feedback and input–output linearization controllers and a nonlinear...... controller based on direct passivation. The comparison is based on time-domain performance and on investigating the stability region, robustness and tuning possibilities of the controllers. Controllers using partial state feedback of the substrate concentration and not directly depending on the reaction rate...... are recommended for the simple fermenter. Passivity based controllers have been found to be globally stable, not very sensitive to the uncertainties in the reaction rate and controller parameter but they require full nonlinear state feedback....

  15. Photonic Crystal Nanocavity Devices for Nonlinear Signal Processing

    DEFF Research Database (Denmark)

    Yu, Yi

    , membranization of InP/InGaAs structure and wet etching. Experimental investigation of the switching dynamics of InP photonic crystal nanocavity structures are carried out using short-pulse homodyne pump-probe techniques, both in the linear and nonlinear region where the cavity is perturbed by a relatively small......This thesis deals with the investigation of InP material based photonic crystal cavity membrane structures, both experimentally and theoretically. The work emphasizes on the understanding of the physics underlying the structures’ nonlinear properties and their applications for all-optical signal...... and large pump power. The experimental results are compared with coupled mode equations developed based on the first order perturbation theory, and carrier rate equations we established for the dynamics of the carrier density governing the cavity properties. The experimental observations show a good...

  16. Neural Generalized Predictive Control of a non-linear Process

    DEFF Research Database (Denmark)

    Sørensen, Paul Haase; Nørgård, Peter Magnus; Ravn, Ole

    1998-01-01

    The use of neural network in non-linear control is made difficult by the fact the stability and robustness is not guaranteed and that the implementation in real time is non-trivial. In this paper we introduce a predictive controller based on a neural network model which has promising stability...... detail and discuss the implementation difficulties. The neural generalized predictive controller is tested on a pneumatic servo sys-tem....

  17. Absorbing Markov Chain Models to Determine Optimum Process Target Levels in Production Systems with Dual Correlated Quality Characteristics

    Directory of Open Access Journals (Sweden)

    Mohammad Saber Fallah Nezhad

    2012-03-01

    Full Text Available For a manufacturing organization to compete effectively in the global marketplace, cutting costs and improving overall efficiency is essential.  A single-stage production system with two independent quality characteristics and different costs associated with each quality characteristic that falls below a lower specification limit (scrap or above an upper specification limit (rework is presented in this paper. The amount of reworks and scraps are assumed to be depending on the process parameters such as process mean and standard deviation thus the expected total profit is significantly dependent on the process parameters. This paper develops a Markovian decision making model for determining the process means. Sensitivity analyzes is performed to validate, and a numerical example is given to illustrate the proposed model. The results showed that the optimal process means extremely effects on the quality characteristics’ parameters.

  18. Logics and Models for Stochastic Analysis Beyond Markov Chains

    DEFF Research Database (Denmark)

    Zeng, Kebin

    , because of the generality of ME distributions, we have to leave the world of Markov chains. To support ME distributions with multiple exits, we introduce a multi-exits ME distribution together with a process algebra MEME to express the systems having the semantics as Markov renewal processes with ME...

  19. Validation of individual-based Markov-like stochastic process model of insect behaviour and a ‘virtual farm’ concept for enhancement of site-specific IPM

    Directory of Open Access Journals (Sweden)

    Slawomir Antoni Lux

    2016-08-01

    Full Text Available The paper reports application of a Markov-like stochastic process agent-based model and a ‘virtual farm’ concept for enhancement of site-specific Integrated Pest Management. Conceptually, the model represents a ‘bottom-up ethological’ approach and emulates behaviour of the ‘primary IPM actors’ - large cohorts of individual insects - within seasonally changing mosaics of spatiotemporally complex faming landscape, under the challenge of the local IPM actions. Algorithms of the proprietary PESTonFARM model were adjusted to reflect behaviour and ecology of R. cerasi. Model parametrization was based on compiled published information about R. cerasi and the results of auxiliary on-farm experiments. The experiments were conducted on sweet cherry farms located in Austria, Germany and Belgium. For each farm, a customised model-module was prepared, reflecting its spatiotemporal features. Historical data about pest monitoring, IPM treatments and fruit infestation were used to specify the model assumptions and calibrate it further. Finally, for each of the farms, virtual IPM experiments were simulated and the model-generated results were compared with the results of the real experiments conducted on the same farms. Implications of the findings for broader applicability of the model and the ‘virtual farm’ approach - were discussed.

  20. Validation of Individual-Based Markov-Like Stochastic Process Model of Insect Behavior and a "Virtual Farm" Concept for Enhancement of Site-Specific IPM.

    Science.gov (United States)

    Lux, Slawomir A; Wnuk, Andrzej; Vogt, Heidrun; Belien, Tim; Spornberger, Andreas; Studnicki, Marcin

    2016-01-01

    The paper reports application of a Markov-like stochastic process agent-based model and a "virtual farm" concept for enhancement of site-specific Integrated Pest Management. Conceptually, the model represents a "bottom-up ethological" approach and emulates behavior of the "primary IPM actors"-large cohorts of individual insects-within seasonally changing mosaics of spatiotemporally complex faming landscape, under the challenge of the local IPM actions. Algorithms of the proprietary PESTonFARM model were adjusted to reflect behavior and ecology of R. cerasi. Model parametrization was based on compiled published information about R. cerasi and the results of auxiliary on-farm experiments. The experiments were conducted on sweet cherry farms located in Austria, Germany, and Belgium. For each farm, a customized model-module was prepared, reflecting its spatiotemporal features. Historical data about pest monitoring, IPM treatments and fruit infestation were used to specify the model assumptions and calibrate it further. Finally, for each of the farms, virtual IPM experiments were simulated and the model-generated results were compared with the results of the real experiments conducted on the same farms. Implications of the findings for broader applicability of the model and the "virtual farm" approach-were discussed.

  1. Quantum Markov Chain Mixing and Dissipative Engineering

    DEFF Research Database (Denmark)

    Kastoryano, Michael James

    2012-01-01

    This thesis is the fruit of investigations on the extension of ideas of Markov chain mixing to the quantum setting, and its application to problems of dissipative engineering. A Markov chain describes a statistical process where the probability of future events depends only on the state of the sy......This thesis is the fruit of investigations on the extension of ideas of Markov chain mixing to the quantum setting, and its application to problems of dissipative engineering. A Markov chain describes a statistical process where the probability of future events depends only on the state...... of the system at the present point in time, but not on the history of events. Very many important processes in nature are of this type, therefore a good understanding of their behaviour has turned out to be very fruitful for science. Markov chains always have a non-empty set of limiting distributions....... Finally, we consider three independent tasks of dissipative engineering: dissipatively preparing a maximally entangled state of two atoms trapped in an optical cavity, dissipative preparation of graph states, and dissipative quantum computing construction....

  2. Real-Time Implementation of Nonlinear Optical Processing Functions.

    Science.gov (United States)

    1986-09-30

    demonstrating that the memory is nonlinear and selective. The recording medium could be replaced with real-time media such as photorefractive crystals. Thicker...recording media Fi4 4. Schematic of experiment that d,.non* trated ,,pera have the added advantage of higher angular selectiv- "" . e e r aity. thus... geometrica snapes in contact ’A,.n a c-:’:ser ’Figure 51a’ ., and a spher:cal 4:verg.ng reference -eam Upion :"um’latlon of t -" c-’gram by the object beam

  3. Decorrelation strategies for the integration of finite sequences of a stochastic process into Gauss-Markov models

    Science.gov (United States)

    Schuh, Wolf-Dieter; Brockmann, Jan Martin; Kargoll, Boris; Loth, Ina

    2014-05-01

    The modeling of satellite measurements series pose a special challenge because of the huge amount of data and the strong correlations between the measurements. In connection with the large number of parameters the rigorous computation of an appropriate stochastic model is a demanding task. This contribution discusses the large variety of possible strategies to treat correlated measurements with constant sampling rate. At first view the regularly structured covariance matrices suggest fast Toeplitz algorithms. But the equidistant measurement series can be also interpreted as a finite sequence of a time discrete covariance stationary stochastic processes with infinite extension. The stochastic process can be represented and analyzed in different quantities in the time domain as well as in the frequency domain. The signal itself and its autocovariance function in the time domain be accompanied by the periodogram and the spectral distribution/spectral density function in the frequency domain. These four quantities and their relations in between can be clearly represented in form of a "Magic Square", which gives a good basis to analyze the stochastic process, study truncation effects and model the behavior by parametric and non parametric approaches. The focus of this study considers the pro and cons of possible strategies to decorrelate finite sequences of measurements and is motivated by GOCE gradiometer measurements. The measurement series are characterized by large correlations over long time spans with a periodic behavior with respect to the orbital period and a fragmentation of the time series into parts, due to satellite maneuvers and calibration phases. The decorrelation process is crucial for the gravity field estimation. Therefore efficient strategies are necessary to get as much signal as possible also from the highly correlated, fragmented measurements series. Special attentions has to take place to avoid data loss during the warmup phase of recursive

  4. An extensive Markov system for ECG exact coding.

    Science.gov (United States)

    Tai, S C

    1995-02-01

    In this paper, an extensive Markov process, which considers both the coding redundancy and the intersample redundancy, is presented to measure the entropy value of an ECG signal more accurately. It utilizes the intersample correlations by predicting the incoming n samples based on the previous m samples which constitute an extensive Markov process state. Theories of the extensive Markov process and conventional n repeated applications of m-th order Markov process are studied first in this paper. After that, they are realized for ECG exact coding. Results show that a better performance can be achieved by our system. The average code length for the extensive Markov system on the second difference signals was 2.512 b/sample, while the average Huffman code length for the second difference signals was 3.326 b/sample.

  5. Approaches to handle nonlinearities and nonnormalities in process chemometrics

    NARCIS (Netherlands)

    Thissen, Uwe Maria Johannes

    2004-01-01

    For every industrial process, it is of paramount interest to online monitor the performance of the process and to assess the quality of the products made. In order to meet these goals, the field of process control works on understanding and improving industrial processes. Process chemometrics can be

  6. A new Markov Binomial distribution

    OpenAIRE

    Leda D. Minkova; Omey, Edward

    2011-01-01

    In this paper, we introduce a two state homogeneous Markov chain and define a geometric distribution related to this Markov chain. We define also the negative binomial distribution similar to the classical case and call it NB related to interrupted Markov chain. The new binomial distribution is related to the interrupted Markov chain. Some characterization properties of the Geometric distributions are given. Recursion formulas and probability mass functions for the NB distribution and the new...

  7. Improving the capability of an integrated CA-Markov model to simulate spatio-temporal urban growth trends using an Analytical Hierarchy Process and Frequency Ratio

    Science.gov (United States)

    Aburas, Maher Milad; Ho, Yuek Ming; Ramli, Mohammad Firuz; Ash'aari, Zulfa Hanan

    2017-07-01

    The creation of an accurate simulation of future urban growth is considered one of the most important challenges in urban studies that involve spatial modeling. The purpose of this study is to improve the simulation capability of an integrated CA-Markov Chain (CA-MC) model using CA-MC based on the Analytical Hierarchy Process (AHP) and CA-MC based on Frequency Ratio (FR), both applied in Seremban, Malaysia, as well as to compare the performance and accuracy between the traditional and hybrid models. Various physical, socio-economic, utilities, and environmental criteria were used as predictors, including elevation, slope, soil texture, population density, distance to commercial area, distance to educational area, distance to residential area, distance to industrial area, distance to roads, distance to highway, distance to railway, distance to power line, distance to stream, and land cover. For calibration, three models were applied to simulate urban growth trends in 2010; the actual data of 2010 were used for model validation utilizing the Relative Operating Characteristic (ROC) and Kappa coefficient methods Consequently, future urban growth maps of 2020 and 2030 were created. The validation findings confirm that the integration of the CA-MC model with the FR model and employing the significant driving force of urban growth in the simulation process have resulted in the improved simulation capability of the CA-MC model. This study has provided a novel approach for improving the CA-MC model based on FR, which will provide powerful support to planners and decision-makers in the development of future sustainable urban planning.

  8. Efficient Modelling and Generation of Markov Automata

    NARCIS (Netherlands)

    Timmer, Mark; Katoen, Joost P.; van de Pol, Jan Cornelis; Stoelinga, Mariëlle Ida Antoinette

    2012-01-01

    This presentation introduces a process-algebraic framework with data for modelling and generating Markov automata. We show how an existing linearisation procedure for process-algebraic representations of probabilistic automata can be reused to transform systems in our new framework to a special

  9. Error exponents for Neyman-Pearson detection of a continuous-time Gaussian Markov process from noisy irregular samples

    CERN Document Server

    Hachem, Walid; Roueff, Francois

    2009-01-01

    This paper addresses the detection of a stochastic process in noise from irregular samples. We consider two hypotheses. The \\emph{noise only} hypothesis amounts to model the observations as a sample of a i.i.d. Gaussian random variables (noise only). The \\emph{signal plus noise} hypothesis models the observations as the samples of a continuous time stationary Gaussian process (the signal) taken at known but random time-instants corrupted with an additive noise. Two binary tests are considered, depending on which assumptions is retained as the null hypothesis. Assuming that the signal is a linear combination of the solution of a multidimensional stochastic differential equation (SDE), it is shown that the minimum Type II error probability decreases exponentially in the number of samples when the False Alarm probability is fixed. This behavior is described by \\emph{error exponents} that are completely characterized. It turns out that they are related with the asymptotic behavior of the Kalman Filter in random s...

  10. Markov chains for testing redundant software

    Science.gov (United States)

    White, Allan L.; Sjogren, Jon A.

    1988-01-01

    A preliminary design for a validation experiment has been developed that addresses several problems unique to assuring the extremely high quality of multiple-version programs in process-control software. The procedure uses Markov chains to model the error states of the multiple version programs. The programs are observed during simulated process-control testing, and estimates are obtained for the transition probabilities between the states of the Markov chain. The experimental Markov chain model is then expanded into a reliability model that takes into account the inertia of the system being controlled. The reliability of the multiple version software is computed from this reliability model at a given confidence level using confidence intervals obtained for the transition probabilities during the experiment. An example demonstrating the method is provided.

  11. A nonlinear optoelectronic filter for electronic signal processing

    Science.gov (United States)

    Loh, William; Yegnanarayanan, Siva; Ram, Rajeev J.; Juodawlkis, Paul W.

    2014-01-01

    The conversion of electrical signals into modulated optical waves and back into electrical signals provides the capacity for low-loss radio-frequency (RF) signal transfer over optical fiber. Here, we show that the unique properties of this microwave-photonic link also enable the manipulation of RF signals beyond what is possible in conventional systems. We achieve these capabilities by realizing a novel nonlinear filter, which acts to suppress a stronger RF signal in the presence of a weaker signal independent of their separation in frequency. Using this filter, we demonstrate a relative suppression of 56 dB for a stronger signal having a 1-GHz center frequency, uncovering the presence of otherwise undetectable weaker signals located as close as 3.5 Hz away. The capabilities of the optoelectronic filter break the conventional limits of signal detection, opening up new possibilities for radar and communication systems, and for the field of precision frequency metrology. PMID:24402418

  12. Nonlinear model predictive control for chemical looping process

    Energy Technology Data Exchange (ETDEWEB)

    Joshi, Abhinaya; Lei, Hao; Lou, Xinsheng

    2017-08-22

    A control system for optimizing a chemical looping ("CL") plant includes a reduced order mathematical model ("ROM") that is designed by eliminating mathematical terms that have minimal effect on the outcome. A non-linear optimizer provides various inputs to the ROM and monitors the outputs to determine the optimum inputs that are then provided to the CL plant. An estimator estimates the values of various internal state variables of the CL plant. The system has one structure adapted to control a CL plant that only provides pressure measurements in the CL loops A and B, a second structure adapted to a CL plant that provides pressure measurements and solid levels in both loops A, and B, and a third structure adapted to control a CL plant that provides full information on internal state variables. A final structure provides a neural network NMPC controller to control operation of loops A and B.

  13. Viscosity Solution of Mean-Variance Portfolio Selection of a Jump Markov Process with No-Shorting Constraints

    Directory of Open Access Journals (Sweden)

    Moussa Kounta

    2016-01-01

    Full Text Available We consider the so-called mean-variance portfolio selection problem in continuous time under the constraint that the short-selling of stocks is prohibited where all the market coefficients are random processes. In this situation the Hamilton-Jacobi-Bellman (HJB equation of the value function of the auxiliary problem becomes a coupled system of backward stochastic partial differential equation. In fact, the value function V often does not have the smoothness properties needed to interpret it as a solution to the dynamic programming partial differential equation in the usual (classical sense; however, in such cases V can be interpreted as a viscosity solution. Here we show the unicity of the viscosity solution and we see that the optimal and the value functions are piecewise linear functions based on some Riccati differential equations. In particular we solve the open problem posed by Li and Zhou and Zhou and Yin.

  14. Design of Tree Structure in Context Clustering Process of Hidden Markov Model-Based Thai Speech Synthesis

    Directory of Open Access Journals (Sweden)

    Suphattharachai Chomphan

    2012-01-01

    Full Text Available Problem statement: In HMM-based Thai speech synthesis, the tone degradation due to the imbalance of training data of all tones. Some distortion of syllable duration is obviously noticeable when the system is trained with a small amount of data. These problems cause the decrement in naturalness and intelligibility of the synthesized speech. Approach: This study proposes an approach to improve the correctness of tone of the synthesized speech which is generated by an HMM-based Thai speech synthesis system. In the tree-based context clustering process, tone groups and tone types are used to design four different structures of decision tree including a single binary tree structure, a simple tone-separated tree structure, a constancy-based-tone-separated tree structure and a trend-based-tone-separated tree structure. Results: A subjective evaluation of tone correctness is conducted by using tone perception of eight Thai listeners. The simple tone-separated tree structure gives the highest level of tone correctness, while the single binary tree structure gives the lowest level of tone correctness. The additional contextual tone information which is applied to all structures of the decision tree achieves a significant improvement of tone correctness. Finally, the evaluation of syllable duration distortion among the four structures shows that the constancy-based-tone-separated and the trend-based-tone-separated tree structures can alleviate the distortions that appear when using the simple tone-separated tree structure. Conclusion: The appropriate structure of tree in context clustering process with the additional contextual tone information can improve the correctness of tones, while the constancy-based-tone-separated and the trend-based-tone-separated tree structures can alleviate the syllable duration distortions.

  15. A Markov Switching Regime Model of Malaysia Property Cycle

    Directory of Open Access Journals (Sweden)

    Abdul M. Beksin

    2011-01-01

    Full Text Available Problem statement: Non-linear models such as the Markov Switching regime (MS method of modelling business cycles, in principle can be used to model property cyle. Approach: The MS model can distinguish property cycle in recession and expansion phases and is sufficiently flexible to allow different relationships to apply over these phases. In this study, the Malaysian property cycle is modelled using a MS model. Results: This technique can be used to simultaneously estimate the data generating process of real GDP growth and classify each observation into one of two regimes (i.e., low-growth and high-growth regimes. Conclusions: This finding has important policy implications, since the yield spread was used to generate the time-varying probabilities of the MS model as well as the recession probabilities of the logit model. In other words, a strong relationship exists between interest rates and the business cycle, where interest rates lead the business cycle.

  16. High-Order Volterra Model Predictive Control and Its Application to a Nonlinear Polymerisation Process

    Institute of Scientific and Technical Information of China (English)

    Yun Li; Hiroshi Kashiwagi

    2005-01-01

    Model Predictive Control (MPC) has recently found wide acceptance in the process industry, but existing design and implementation methods are restricted to linear process models. A chemical process, however, involves severe nonlinearity which cannot be ignored in practice. This paper aims to solve this nonlinear control problem by extending MPC to accommodate nonlinear models. It develops an analytical framework for nonlinear model predictive control (NMPC). It also offers a third-order Volterra series based nonparametric nonlinear modelling technique for NMPC design, which relieves practising engineers from the need for deriving a physical-principles based model first. An on-line realisation technique for implementing NMPC is then developed and applied to a Mitsubishi Chemicals polymerisation reaction process. Results show that this nonlinear MPC technique is feasible and very effective. It considerably outperforms linear and low-order Volterra model based methods. The advantages of the developed approach lie not only in control performance superior to existing NMPC methods, but also in eliminating the need for converting an analytical model and then convert it to a Volterra model obtainable only up to the second order.

  17. Partially Hidden Markov Models

    DEFF Research Database (Denmark)

    Forchhammer, Søren Otto; Rissanen, Jorma

    1996-01-01

    Partially Hidden Markov Models (PHMM) are introduced. They differ from the ordinary HMM's in that both the transition probabilities of the hidden states and the output probabilities are conditioned on past observations. As an illustration they are applied to black and white image compression wher...

  18. On Weak Markov's Principle

    DEFF Research Database (Denmark)

    Kohlenbach, Ulrich Wilhelm

    2002-01-01

    We show that the so-called weak Markov's principle (WMP) which states that every pseudo-positive real number is positive is underivable in E-HA + AC. Since allows one to formalize (atl eastl arge parts of) Bishop's constructive mathematics, this makes it unlikely that WMP can be proved within the...

  19. Optical pulse engineering and processing using optical nonlinearities of nanostructured waveguides made of silicon

    Science.gov (United States)

    Lavdas, Spyros; You, Jie; Osgood, Richard M.; Panoiu, Nicolae C.

    2015-08-01

    We present recent results pertaining to pulse reshaping and optical signal processing using optical nonlinearities of silicon-based tapered photonic wires and photonic crystal waveguides. In particular, we show how nonlinearity and dispersion engineering of tapered photonic wires can be employed to generate optical similaritons and achieve more than 10× pulse compression. We also discuss the properties of four-wave mixing pulse amplification and frequency conversion efficiency in long-period Bragg waveguides and photonic crystal waveguides. Finally, the influence of linear and nonlinear optical effects on the transmission bit-error rate in uniform photonic wires and photonic crystal waveguides made of silicon is discussed.

  20. Compositional Modeling and Minimization of Time-Inhomogeneous Markov Chains

    NARCIS (Netherlands)

    Han, T.; Katoen, J.P.; Mereacre, A.

    2008-01-01

    This paper presents a compositional framework for the modeling of interactive continuous-time Markov chains with time-dependent rates, a subclass of communicating piecewise deterministic Markov processes. A poly-time algorithm is presented for computing the coarsest quotient under strong bisimulatio

  1. On the Markovian assumption in the excursion set approach: The approximation of Markov Velocities

    CERN Document Server

    Musso, Marcello

    2014-01-01

    The excursion set approach uses the statistics of the density field, smoothed on a wide range of scales, to gain insight into a number of interesting processes in nonlinear structure formation, such as cluster assembly, merging and clustering. The approach treats the curve defined by the overdensity fluctuation field when changing the smoothing scale as a random walk. Most implementations of the approach then assume that, at least to a first approximation, the walks have uncorrelated steps, so that the walk heights are a Markov process. This assumption is known to be inaccurate: smoothing filters that are most easily related to the physics of structure formation generically yield walks whose steps are correlated with one another. We develop models in which it is the steps, rather than the walk heights, that are a Markov process. In such models, which we call Markov Velocity processes, each step correlates only with the previous one. We show that TopHat smoothing of a power law power spectrum with index n = -2...

  2. Blind Image Deblurring Driven by Nonlinear Processing in the Edge Domain

    Directory of Open Access Journals (Sweden)

    Stefania Colonnese

    2004-12-01

    Full Text Available This work addresses the problem of blind image deblurring, that is, of recovering an original image observed through one or more unknown linear channels and corrupted by additive noise. We resort to an iterative algorithm, belonging to the class of Bussgang algorithms, based on alternating a linear and a nonlinear image estimation stage. In detail, we investigate the design of a novel nonlinear processing acting on the Radon transform of the image edges. This choice is motivated by the fact that the Radon transform of the image edges well describes the structural image features and the effect of blur, thus simplifying the nonlinearity design. The effect of the nonlinear processing is to thin the blurred image edges and to drive the overall blind restoration algorithm to a sharp, focused image. The performance of the algorithm is assessed by experimental results pertaining to restoration of blurred natural images.

  3. Data Analysis Techniques for Resolving Nonlinear Processes in Plasmas : a Review

    OpenAIRE

    de Wit, T. Dudok

    1996-01-01

    The growing need for a better understanding of nonlinear processes in plasma physics has in the last decades stimulated the development of new and more advanced data analysis techniques. This review lists some of the basic properties one may wish to infer from a data set and then presents appropriate analysis techniques with some recent applications. The emphasis is put on the investigation of nonlinear wave phenomena and turbulence in space plasmas.

  4. 加权分支过程的单调性、对偶性和Feller性质%The Monotonicity, Duality and Feller Property of Weighted Markov Branching Process

    Institute of Scientific and Technical Information of China (English)

    丁水琴; 李扬荣

    2008-01-01

    In this paper, the monotonicity, duality and Feller property of weighted Markov branching processes are studied and some necessary and sufficient conditions for the minimal Q-function being an monotone or dual transition function are obtained, where Q is a weighted Markov branching q-matrix. Especially, Feller criteria are obtained when Q is neither dual nor monotone.%研究加权分支过程的单调性, 对偶性以及Feller性质, 并得到了加权分支q矩阵的最小Q函数成为单调或对偶时的充要条件, 特别是得到了当Q既不对偶也不单调时的Feller准则.

  5. 马尔可夫更新过程在武器装备可靠性方面的应用%Application of Markov Renewal Process in the Credibility Aspect of Weapon Equipment

    Institute of Scientific and Technical Information of China (English)

    余超; 陈夫凯; 周浩

    2014-01-01

    区别于单一武器装备寿命分布分析,考虑综合性情况下的非马尔可夫型冷储备可修系统,应用马尔可夫更新过程理论建立可靠性分析模型,并计算出武器装备的2种可用度。%To make a distinction from the analysis of life span distribution about the single weapon e-quipment,the paper considered the non-Markov in comprehensive situation,used the Markov renewal process theory to build the reliability analysis model,and calculated the two kinds of usability.

  6. On the entropy rate and mutual information rate of Markov modulated Poisson process%马尔科夫调制泊松点过程的熵率和相互信息率∗

    Institute of Scientific and Technical Information of China (English)

    吕绍川

    2015-01-01

    显示地导出了一类连续时间参数隐马尔科夫模型-马尔科夫调制泊松点过程(MMPP)的熵率和相互信息率。模拟研究表明这类隐马尔科夫模型参数的最大似然估计的精度和效与观测过程和隐过程之间的相互信息率密切相关。一般地,相互信息率可作为 MMPP 中各个混合分量广义距离(或差别性)的一个度量。%The Entropy rate and mutual information rate of a hidden Markov model,the Markov modulated Poisson process (MMPP),are explicitly derived.Simulation studies suggest that the accuracy and efficiency of Maximum Likelihood Estimation (MLE)of this class of models are closely associated with mutual information rate between observed point process and latent Markov chain.

  7. Simulations of the Ocean Response to a Hurricane: Nonlinear Processes

    KAUST Repository

    Zedler, Sarah E.

    2009-10-01

    Superinertial internal waves generated by a tropical cyclone can propagate vertically and laterally away from their local generation site and break, contributing to turbulent vertical mixing in the deep ocean and maintenance of the stratification of the main thermocline. In this paper, the results of a modeling study are reported to investigate the mechanism by which superinertial fluctuations are generated in the deep ocean. The general properties of the superinertial wave wake were also characterized as a function of storm speed and central latitude. The Massachusetts Institute of Technology (MIT) Ocean General Circulation Model (OGCM) was used to simulate the open ocean response to realistic westward-tracking hurricane-type surface wind stress and heat and net freshwater buoyancy forcing for regions representative of midlatitudes in the Atlantic, the Caribbean, and low latitudes in the eastern Pacific. The model had high horizontal [Δ(x, y) = 1/6°] and vertical (Δz = 5 m in top 100 m) resolution and employed a parameterization for vertical mixing induced by shear instability. In the horizontal momentum equation, the relative size of the nonlinear advection terms, which had a dominant frequency near twice the inertial, was large only in the upper 200 m of water. Below 200 m, the linear momentum equations obeyed a linear balance to 2%. Fluctuations at nearly twice the inertial frequency (2f) were prevalent throughout the depth of the water column, indicating that these nonlinear advection terms in the upper 200 m forced a linear mode below at nearly twice the inertial frequency via vorticity conservation. Maximum variance at 2f in horizontal velocity occurred on the south side of the track. This was in response to vertical advection of northward momentum, which in the north momentum equation is an oscillatory positive definite term that constituted a net force to the south at a frequency near 2f. The ratio of this term to the Coriolis force was larger on the

  8. Nonlinearities in the quantum measurement process of superconducting qubits

    Energy Technology Data Exchange (ETDEWEB)

    Serban, Ioana

    2008-05-15

    The work described in this thesis focuses on the investigation of decoherence and measurement backaction, on the theoretical description of measurement schemes and their improvement. The study presented here is centered around quantum computing implementations using superconducting devices and most important, the Josephson effect. The measured system is invariantly a qubit, i. e. a two-level system. The objective is to study detectors with increasing nonlinearity, e. g. coupling of the qubit to the frequency a driven oscillator, or to the bifurcation amplifier, to determine the performance and backaction of the detector on the measured system and to investigate the importance of a strong qubit-detector coupling for the achievement of a quantum non-demolition type of detection. The first part gives a very basic introduction to quantum information, briefly reviews some of the most promising physical implementations of a quantum computer before focusing on the superconducting devices. The second part presents a series of studies of different qubit measurements, describing the backaction of the measurement onto the measured system and the internal dynamics of the detector. Methodology adapted from quantum optics and chemical physics (master equations, phase-space analysis etc.) combined with the representation of a complex environment yielded a tool capable of describing a nonlinear, non-Markovian environment, which couples arbitrarily strongly to the measured system. This is described in chapter 3. Chapter 4 focuses on the backaction on the qubit and presents novel insights into the qubit dephasing in the strong coupling regime. Chapter 5 uses basically the same system and technical tools to explore the potential of a fast, strong, indirect measurement, and determine how close such a detection would ideally come to the quantum non-demolition regime. Chapter 6 focuses on the internal dynamics of a strongly driven Josephson junction. The analytical results are based on

  9. Correntropy-based partial directed coherence for testing multivariate Granger causality in nonlinear processes

    Science.gov (United States)

    Kannan, Rohit; Tangirala, Arun K.

    2014-06-01

    Identification of directional influences in multivariate systems is of prime importance in several applications of engineering and sciences such as plant topology reconstruction, fault detection and diagnosis, and neurosciences. A spectrum of related directionality measures, ranging from linear measures such as partial directed coherence (PDC) to nonlinear measures such as transfer entropy, have emerged over the past two decades. The PDC-based technique is simple and effective, but being a linear directionality measure has limited applicability. On the other hand, transfer entropy, despite being a robust nonlinear measure, is computationally intensive and practically implementable only for bivariate processes. The objective of this work is to develop a nonlinear directionality measure, termed as KPDC, that possesses the simplicity of PDC but is still applicable to nonlinear processes. The technique is founded on a nonlinear measure called correntropy, a recently proposed generalized correlation measure. The proposed method is equivalent to constructing PDC in a kernel space where the PDC is estimated using a vector autoregressive model built on correntropy. A consistent estimator of the KPDC is developed and important theoretical results are established. A permutation scheme combined with the sequential Bonferroni procedure is proposed for testing hypothesis on absence of causality. It is demonstrated through several case studies that the proposed methodology effectively detects Granger causality in nonlinear processes.

  10. Solving method of generalized nonlinear dynamic least squares for data processing in building of digital mine

    Institute of Scientific and Technical Information of China (English)

    TAO Hua-xue (陶华学); GUO Jin-yun (郭金运)

    2003-01-01

    Data are very important to build the digital mine. Data come from many sources, have different types and temporal states. Relations between one class of data and the other one, or between data and unknown parameters are more nonlinear. The unknown parameters are non-random or random, among which the random parameters often dynamically vary with time. Therefore it is not accurate and reliable to process the data in building the digital mine with the classical least squares method or the method of the common nonlinear least squares. So a generalized nonlinear dynamic least squares method to process data in building the digital mine is put forward. In the meantime, the corresponding mathematical model is also given. The generalized nonlinear least squares problem is more complex than the common nonlinear least squares problem and its solution is more difficultly obtained because the dimensions of data and parameters in the former are bigger. So a new solution model and the method are put forward to solve the generalized nonlinear dynamic least squares problem. In fact, the problem can be converted to two sub-problems, each of which has a single variable. That is to say, a complex problem can be separated and then solved. So the dimension of unknown parameters can be reduced to its half, which simplifies the original high dimensional equations. The method lessens the calculating load and opens up a new way to process the data in building the digital mine, which have more sources, different types and more temporal states.

  11. Parallel Markov chain Monte Carlo simulations.

    Science.gov (United States)

    Ren, Ruichao; Orkoulas, G

    2007-06-07

    With strict detailed balance, parallel Monte Carlo simulation through domain decomposition cannot be validated with conventional Markov chain theory, which describes an intrinsically serial stochastic process. In this work, the parallel version of Markov chain theory and its role in accelerating Monte Carlo simulations via cluster computing is explored. It is shown that sequential updating is the key to improving efficiency in parallel simulations through domain decomposition. A parallel scheme is proposed to reduce interprocessor communication or synchronization, which slows down parallel simulation with increasing number of processors. Parallel simulation results for the two-dimensional lattice gas model show substantial reduction of simulation time for systems of moderate and large size.

  12. Markov bridges, bisection and variance reduction

    DEFF Research Database (Denmark)

    Asmussen, Søren; Hobolth, Asger

    Time-continuous Markov jump processes is a popular modelling tool in disciplines ranging from computational finance and operations research to human genetics and genomics. The data is often sampled at discrete points in time, and it can be useful to simulate sample paths between the datapoints....... In this paper we firstly consider the problem of generating sample paths from a continuous-time Markov chain conditioned on the endpoints using a new algorithm based on the idea of bisection. Secondly we study the potential of the bisection algorithm for variance reduction. In particular, examples are presented...... where the methods of stratification, importance sampling and quasi Monte Carlo are investigated....

  13. 研究M/G/1排队的一个新的向量马氏过程%A New Vector Markov Process for M/G/1 Queue

    Institute of Scientific and Technical Information of China (English)

    严庆强; 史定华; 郭兴国

    2005-01-01

    In this paper, by considering the stochastic process of the busy period and the idle period, and introducing the unfinished work as a supplementary variable, a new vector Markov process was presented to study the M/G/1 queue again. Through establishing and solving the density evolution equations, the busy-period distribution, and the stationary distributions of waiting time and queue length were obtained. In addition, the stability condition of this queue system was given by means of an imbedded renewal process.

  14. Regularity of harmonic functions for some Markov chains with unbounded range

    OpenAIRE

    2012-01-01

    We consider a class of continuous time Markov chains on $\\Z^d$. These chains are the discrete space analogue of Markov processes with jumps. Under some conditions, we show that harmonic functions associated with these Markov chains are H\\"{o}lder continuous.

  15. Nonlinear enhancement in photonic crystal slow light waveguides fabricated using CMOS-compatible process.

    Science.gov (United States)

    Shinkawa, Mizuki; Ishikura, Norihiro; Hama, Yosuke; Suzuki, Keijiro; Baba, Toshihiko

    2011-10-24

    We have studied low-dispersion slow light and its nonlinear enhancement in photonic crystal waveguides. In this work, we fabricated the waveguides using Si CMOS-compatible process. It enables us to integrate spotsize converters, which greatly simplifies the optical coupling from fibers as well as demonstration of the nonlinear enhancement. Two-photon absorption, self-phase modulation and four-wave mixing were observed clearly for picosecond pulses in a 200-μm-long device. In comparison with Si wire waveguides, a 60-120 fold higher nonlinearity was evaluated for a group index of 51. Unique intensity response also occurred due to the specific transmission spectrum and enhanced nonlinearities. Such slow light may add various functionalities in Si photonics, while loss reduction is desired for ensuring the advantage of slow light.

  16. Nonlinear Transport Processes in Tokamak Plasmas. Part I: The Collisional Regimes

    CERN Document Server

    Sonnino, Giorgio

    2008-01-01

    An application of the thermodynamic field theory (TFT) to transport processes in L-mode tokamak plasmas is presented. The nonlinear corrections to the linear (Onsager) transport coefficients in the collisional regimes are derived. A quite encouraging result is the appearance of an asymmetry between the Pfirsch-Schlueter (P-S) ion and electron transport coefficients: the latter presents a nonlinear correction, which is absent for the ions, and makes the radial electron coefficients much larger than the former. Explicit calculations and comparisons between the neoclassical results and the TFT predictions for JET plasmas are also reported. We found that the nonlinear electron P-S transport coefficients exceed the values provided by neoclassical theory by a factor, which may be of the order 100. The nonlinear classical coefficients exceed the neoclassical ones by a factor, which may be of order 2. The expressions of the ion transport coefficients, determined by the neoclassical theory in these two regimes, remain...

  17. Polarization dependent dispersion and its impact on optical parametric process in high nonlinear microstructure fibre

    Institute of Scientific and Technical Information of China (English)

    Xiao Li; Zhang Wei; Huang Yi-Dong; Peng Jiang-De

    2008-01-01

    High nonlinear microstructure fibre (HNMF) is preferred in nonlinear fibre optics, especially in the applications of optical parametric effects, due to its high optical nonlinear coefficient. However, polarization dependent dispersion will impact the nonlinear optical parametric process in HNMFs. In this paper, modulation instability (MI) method is used to measure the polarization dependent dispersion of a piece of commercial HNMF, including the group velocity dispersion, the dispersion slope, the fourth-order dispersion and group birefringence. It also experimentally demonstrates the impact of the polarization dependent dispersion on the continuous wave supercontinuum (SC) generation. On one axis MI sidebands with symmetric frequency dctunings are generated, while on the other axis with larger MI frequency detuning, SC is generated by soliton self-frequency shift.

  18. Innovation as a nonlinear process, the scientometric perspective, and the specification of an 'innovation opportunities explorer'

    NARCIS (Netherlands)

    Leydesdorff, L.; Rotolo, D.; de Nooy, W.

    2013-01-01

    The process of innovation follows nonlinear patterns across the domains of science, technology, and the economy. Novel bibliometric mapping techniques can be used to investigate and represent distinctive, but complementary perspectives on the innovation process (e.g. ‘demand’ and ‘supply’) as well

  19. 马尔可夫调制的跳扩散过程下幂式期权的定价%Pricing Power Options under a Markov-modulated Jump Diffusion Process

    Institute of Scientific and Technical Information of China (English)

    赵奇杰; 王伟

    2013-01-01

    The conditions of market economy are described by a two-state continuous time Markov chain with a Markov-modulated jump diffusion process satisfied by the risky asset. The pricing problem of power option is considered under a Markov-modulated model. The value formula of power call option is obtained by measuring the change and Girsanov’s theorem, and the value formula of power put option by put-call-parity. The numeric results are also provided using the Monte Carlo simulation technique.%假定市场经济状态由两状态连续时间马尔可夫链描述,风险资产满足马尔可夫调制的跳扩散过程,研究了马尔可夫调制模型下幂式期权的定价问题。通过测度变换和Girsanov定理,得出幂式看涨期权定价公式,并利用看涨、看跌的平价关系得到了幂式看跌期权的定价公式。此外,还利用蒙特卡洛方法给出了幂式看涨期权价值的数值结果。

  20. Confluence reduction for Markov automata (extended version)

    NARCIS (Netherlands)

    Timmer, M.; Pol, van de J.C.; Stoelinga, M.I.A.

    2013-01-01

    Markov automata are a novel formalism for specifying systems exhibiting nondeterminism, probabilistic choices and Markovian rates. Recently, the process algebra MAPA was introduced to efficiently model such systems. As always, the state space explosion threatens the analysability of the models gener

  1. Efficient Modelling and Generation of Markov Automata

    NARCIS (Netherlands)

    Timmer, Mark; Katoen, Joost-Pieter; Pol, van de Jaco; Stoelinga, Mariëlle; Koutny, M.; Ulidowski, I.

    2012-01-01

    This paper introduces a framework for the efficient modelling and generation of Markov automata. It consists of (1) the data-rich process-algebraic language MAPA, allowing concise modelling of systems with nondeterminism, probability and Markovian timing; (2) a restricted form of the language, the M

  2. A Martingale Decomposition of Discrete Markov Chains

    DEFF Research Database (Denmark)

    Hansen, Peter Reinhard

    We consider a multivariate time series whose increments are given from a homogeneous Markov chain. We show that the martingale component of this process can be extracted by a filtering method and establish the corresponding martingale decomposition in closed-form. This representation is useful...

  3. MCMC for non-linear state space models using ensembles of latent sequences

    OpenAIRE

    2013-01-01

    Non-linear state space models are a widely-used class of models for biological, economic, and physical processes. Fitting these models to observed data is a difficult inference problem that has no straightforward solution. We take a Bayesian approach to the inference of unknown parameters of a non-linear state model; this, in turn, requires the availability of efficient Markov Chain Monte Carlo (MCMC) sampling methods for the latent (hidden) variables and model parameters. Using the ensemble ...

  4. Quasilinear Extreme Learning Machine Model Based Internal Model Control for Nonlinear Process

    Directory of Open Access Journals (Sweden)

    Dazi Li

    2015-01-01

    Full Text Available A new strategy for internal model control (IMC is proposed using a regression algorithm of quasilinear model with extreme learning machine (QL-ELM. Aimed at the chemical process with nonlinearity, the learning process of the internal model and inverse model is derived. The proposed QL-ELM is constructed as a linear ARX model with a complicated nonlinear coefficient. It shows some good approximation ability and fast convergence. The complicated coefficients are separated into two parts. The linear part is determined by recursive least square (RLS, while the nonlinear part is identified through extreme learning machine. The parameters of linear part and the output weights of ELM are estimated iteratively. The proposed internal model control is applied to CSTR process. The effectiveness and accuracy of the proposed method are extensively verified through numerical results.

  5. A fuzzy model based adaptive PID controller design for nonlinear and uncertain processes.

    Science.gov (United States)

    Savran, Aydogan; Kahraman, Gokalp

    2014-03-01

    We develop a novel adaptive tuning method for classical proportional-integral-derivative (PID) controller to control nonlinear processes to adjust PID gains, a problem which is very difficult to overcome in the classical PID controllers. By incorporating classical PID control, which is well-known in industry, to the control of nonlinear processes, we introduce a method which can readily be used by the industry. In this method, controller design does not require a first principal model of the process which is usually very difficult to obtain. Instead, it depends on a fuzzy process model which is constructed from the measured input-output data of the process. A soft limiter is used to impose industrial limits on the control input. The performance of the system is successfully tested on the bioreactor, a highly nonlinear process involving instabilities. Several tests showed the method's success in tracking, robustness to noise, and adaptation properties. We as well compared our system's performance to those of a plant with altered parameters with measurement noise, and obtained less ringing and better tracking. To conclude, we present a novel adaptive control method that is built upon the well-known PID architecture that successfully controls highly nonlinear industrial processes, even under conditions such as strong parameter variations, noise, and instabilities.

  6. Non-linear thermodynamic laws application to soil processes

    Directory of Open Access Journals (Sweden)

    Ilgiz Khabirov

    2013-01-01

    Full Text Available An attempt has been made to analyze the possibility to use nonequilibrium thermodynamics for the soil dynamic open systemstreatment. Entropy change of such a system and the entropy coming from or going into the outer sphere. In the steady state, dynamic soil-formation processes occur within an organized structure and are characterized by stable parameters close to equilibrium. Accordingly, when examining soil, one can proceed from the conventional thermodynamic equilibrium. However, the matter of Onzager-Prigozhin general phenomenological theory applicability to soil processes is more complicated. To study soil stability it is necessary to go beyond the limits of linear thermodynamics.

  7. Nonlinear processes upon two-photon interband picosecond excitation of PbWO4 crystal

    Science.gov (United States)

    Lukanin, V. I.; Karasik, A. Ya

    2016-09-01

    A new experimental method is proposed to study the dynamics of nonlinear processes occurring upon two-photon interband picosecond excitation of a lead tungstate crystal and upon its excitation by cw probe radiation in a temporal range from several nanoseconds to several seconds. The method is applied to the case of crystal excitation by a sequence of 25 high-power picosecond pulses with a wavelength of 523.5 nm and 633-nm cw probe radiation. Measuring the probe beam transmittance during crystal excitation, one can investigate the influence of two-photon interband absorption and the thermal nonlinearity of the refractive index on the dynamics of nonlinear processes in a wide range of times (from several nanoseconds to several seconds). The time resolution of the measuring system makes it possible to distinguish fast and slow nonlinear processes of electronic or thermal nature, including the generation of a thermal lens and thermal diffusion. An alternative method is proposed to study the dynamics of induced absorption transformation and, therefore, the dynamics of the development of nonlinear rocesses upon degenerate two-photon excitation of the crystal in the absence of external probe radiation.

  8. Nonlinear Optical Signal Processing for Tbit/s Ethernet Applications

    DEFF Research Database (Denmark)

    Oxenløwe, Leif Katsuo; Galili, Michael; Mulvad, Hans Christian Hansen;

    2012-01-01

    We review recent experimental demonstrations of Tbaud optical signal processing. In particular, we describe a successful 1.28 Tbit/s serial data generation based on single polarization 1.28 Tbaud symbol rate pulses with binary data modulation (OOK) and subsequent all-optical demultiplexing. We also...

  9. Linear and nonlinear optical processing of polymer matrix nanocomposites

    Science.gov (United States)

    DeJournett, Travis J.; Han, Karen; Olasov, Lauren R.; Zeng, Fan W.; Lee, Brennan; Spicer, James B.

    2015-08-01

    This work focuses on the scalable synthesis and processing of nanostructures in polymer matrix nanocomposites (PMNCs) for applications that require photochemical functionality of these nanostructures. An in situ vapor deposition process using various metal and metal oxide precursors has been used to create a range of nanocomposites that display photochromic and photocatalytic behaviors. Under specific processing conditions, these composites consist of discrete nanoparticles distributed uniformly throughout the bulk of an optically transparent polymer matrix. Incorporating other chemical species as supplementary deposition agents in the synthesis process can modify these particles and produce complicated nanostructures with enhanced properties. In particular, work has been carried out to structure nanoparticles using laser irradiation. Starting with metallic or metal oxide nanoparticles in the polymer matrix, localized chemical vapor deposition in the near-particle environment has been carried out using laser irradiation to decompose chemical precursors leading to the formation of secondary structures surrounding the seed nanoparticles. Control of the spatial and temporal characteristics of the excitation source allows for synthesis of nanocomposites with a high degree of control over the location, composition and size of nanoparticles in the matrix and presents the opportunity to produce patterned materials with spatially varying properties.

  10. A Kernel Time Structure Independent Component Analysis Method for Nonlinear Process Monitoring☆

    Institute of Scientific and Technical Information of China (English)

    Lianfang Cai; Xuemin Tian; Ni Zhang

    2014-01-01

    Kernel independent component analysis (KICA) is a newly emerging nonlinear process monitoring method, which can extract mutually independent latent variables cal ed independent components (ICs) from process var-iables. However, when more than one IC have Gaussian distribution, it cannot extract the IC feature effectively and thus its monitoring performance will be degraded drastical y. To solve such a problem, a kernel time struc-ture independent component analysis (KTSICA) method is proposed for monitoring nonlinear process in this paper. The original process data are mapped into a feature space nonlinearly and then the whitened data are calculated in the feature space by the kernel trick. Subsequently, a time structure independent component analysis algorithm, which has no requirement for the distribution of ICs, is proposed to extract the IC feature. Finally, two monitoring statistics are built to detect process faults. When some fault is detected, a nonlinear fault identification method is developed to identify fault variables based on sensitivity analysis. The proposed monitoring method is applied in the Tennessee Eastman benchmark process. Applications demonstrate the superiority of KTSICA over KICA.

  11. Nonlinear Statistical Process Monitoring Based on Control Charts with Memory Effect and Kernel Independent Component Analysis

    Institute of Scientific and Technical Information of China (English)

    2007-01-01

    A novel nonlinear combination process monitoring method was proposed based on techniques with memory effect (multivariate exponentially weighted moving average (MEWMA)) and kernel independent component analysis (KICA). The method was developed for dealing with nonlinear issues and detecting small or moderate drifts in one or more process variables with autocorrelation. MEWMA charts use additional information from the past history of the process for keeping the memory effect of the process behavior trend. KICA is a recently developed statistical technique for revealing hidden, nonlinear statistically independent factors that underlie sets of measurements and it is a two-phase algorithm: whitened kernel principal component analysis (KPCA) plus independent component analysis (ICA). The application to the fluid catalytic cracking unit (FCCU) simulated process indicates that the proposed combined method based on MEWMA and KICA can effectively capture the nonlinear relationship and detect small drifts in process variables. Its performance significantly outperforms monitoring method based on ICA, MEWMA-ICA and KICA, especially for long-term performance deterioration.

  12. Electrically controlled nonlinear optical generation and signal processing in plasmonic metamaterials (Conference Presentation)

    Science.gov (United States)

    Cai, Wenshan

    2016-09-01

    Metamaterials have offered not only the unprecedented opportunity to generate unconventional electromagnetic properties that are not found in nature, but also the exciting potential to create customized nonlinear media with tailored high-order effects. Two particularly compelling directions of current interests are active metamaterials, where the optical properties can be purposely manipulated by external stimuli, and nonlinear metamaterials, which enable intensity-dependent frequency conversion of light. By exploring the interaction of these two directions, we leverage the electrical and optical functions simultaneously supported in nanostructured metals and demonstrate electrically-controlled nonlinear processes from photonic metamaterials. We show that a variety of nonlinear optical phenomena, including the wave mixing and the optical rectification, can be purposely modulated by applied voltage signals. In addition, electrically-induced and voltage-controlled nonlinear effects facilitate us to demonstrate the backward phase matching in a negative index material, a long standing prediction in nonlinear metamaterials. Other results to be covered in this talk include photon-drag effect in plasmonic metamaterials and ion-assisted nonlinear effects from metamaterials in electrolytes. Our results reveal a grand opportunity to exploit optical metamaterials as self-contained, dynamic electrooptic systems with intrinsically embedded electrical functions and optical nonlinearities. Reference: L. Kang, Y. Cui, S. Lan, S. P. Rodrigues, M. L. Brongersma, and W. Cai, Nature Communications, 5, 4680 (2014). S. P. Rodrigues and W.Cai, Nature Nanotechnology, 10, 387 (2015). S. Lan, L. Kang, D. T. Schoen, S. P. Rodrigues, Y. Cui, M. L. Brongersma, and W. Cai, Nature Materials, 14, 807 (2015).

  13. The interaction of climate observation, parameter estimation, and mitigation decisions: Modeling climate policy under uncertainty with a partially observable Markov decision process

    Science.gov (United States)

    Fertig, E.; Webster, M.

    2013-12-01

    Though climate sensitivity remains poorly constrained, the trajectory of future greenhouse gas emissions and observable climate data could lead to improved estimates. Updated parameter estimates could alter decisions on greenhouse mitigation policy, which in turn influences future observed climate data and parameter estimation. Previous research on global climate mitigation policy neglects the cyclic nature of climate observation, parameter estimation, and policy action, instead treating uncertainty in climate sensitivity with scenario analysis or assuming that it will be resolved completely at some point in the future. This paper advances quantitative analysis of decision making under uncertainty (DMUU) in climate sensitivity by modeling the observation/parameter estimation/policy action cycle as a partially observable Markov decision process (POMDP). In a POMDP framework, an objective function is maximized while both observable parameters and probability distributions over unobservable parameters are retained as system states. As time progresses and more data are collected, the probability distributions are updated with Bayesian analysis. To model anthropogenic climate change as a POMDP, we maximize social welfare using a modified DICE model. Climate sensitivity is never directly observable; instead it is modeled with a distribution that is subject to Bayesian updating after observation of stochastic changes in global mean temperature. The maximization problem is posed as a stochastic Bellman equation, which expresses total social welfare as the sum of immediate social welfare resulting from a current mitigation decision under current knowledge of climate sensitivity and the expected cost-to-go, which is the discounted future social welfare in the subsequent time interval as a function of both global mean temperature and the consequent probability distribution over climate sensitivity. While similar, smaller stochastic dynamic programming problems can be solved

  14. The effect of process delay on dynamical behaviors in a self-feedback nonlinear oscillator

    Science.gov (United States)

    Yao, Chenggui; Ma, Jun; Li, Chuan; He, Zhiwei

    2016-10-01

    The delayed feedback loops play a crucial role in the stability of dynamical systems. The effect of process delay in feedback is studied numerically and theoretically in the delayed feedback nonlinear systems including the neural model, periodic system and chaotic oscillator. The process delay is of key importance in determining the evolution of systems, and the rich dynamical phenomena are observed. By introducing a process delay, we find that it can induce bursting electric activities in the neural model. We demonstrate that this novel regime of amplitude death also exists in the parameter space of feedback strength and process delay for the periodic system and chaotic oscillator. Our results extend the effect of process delay in the paper of Zou et al.(2013) where the process delay can eliminate the amplitude death of the coupled nonlinear systems.

  15. Nonlinear signal processing of electroencephalograms for automated sleep monitoring

    Science.gov (United States)

    Wilson, D.; Rowlands, D. D.; James, Daniel A.; Cutmore, T.

    2005-02-01

    An automated classification technique is desirable to identify the different stages of sleep. In this paper a technique for differentiating the characteristics of each sleep phase has been developed. This is an ideal pre-processor stage for classifying systems such as neural networks. A wavelet based continuous Morlet transform was developed to analyse the EEG signal in both the time and frequency domain. Test results using two 100 epoch EEG test data sets from pre-recorded EEG data are presented. Key rhythms in the EEG signal were identified and classified using the continuous wavelet transform. The wavelet results indicated each sleep phase contained different rhythms and artefacts (noise from muscle movement in the EEG); providing proof that an EEG can be classified accordingly. The coefficients founded by the wavelet transform have been emphasised by statistical techniques. Hypothesis testing was used to highlight major differences between adjacent sleep stages. Various signal processing methods such as power spectrum density and the discrete wavelet transform have been used to emphasise particular characteristics in an EEG. By implementing signal processing methods on an EEG data set specific rules for each sleep stage have been developed suitable for a neural network classification solution.

  16. Non-linear, adaptive array processing for acoustic interference suppression.

    Science.gov (United States)

    Hoppe, Elizabeth; Roan, Michael

    2009-06-01

    A method is introduced where blind source separation of acoustical sources is combined with spatial processing to remove non-Gaussian, broadband interferers from space-time displays such as bearing track recorder displays. This differs from most standard techniques such as generalized sidelobe cancellers in that the separation of signals is not done spatially. The algorithm performance is compared to adaptive beamforming techniques such as minimum variance distortionless response beamforming. Simulations and experiments using two acoustic sources were used to verify the performance of the algorithm. Simulations were also used to determine the effectiveness of the algorithm under various signal to interference, signal to noise, and array geometry conditions. A voice activity detection algorithm was used to benchmark the performance of the source isolation.

  17. Age and Creative Productivity: Nonlinear Estimation of an Information-Processing Model.

    Science.gov (United States)

    Simonton, Dean Keith

    1989-01-01

    Applied two-step cognitive model to relationship between age and creative productivity. Selected ideation and elaboration rates as information-processing parameters that define mathematical function which describes age curves and specifies their variance across disciplines. Applied non-linear estimation program to further validate model. Despite…

  18. Ultrafast nonlinear all-optical processes in silicon-on-insulator waveguides

    NARCIS (Netherlands)

    Dekker, R.; Usechak, N.; Först, M.; Driessen, A.

    2007-01-01

    In this review we present an overview of the progress made in recent years in the field of integrated silicon-on-insulator (SOI) waveguide photonics with a strong emphasis on third-order nonlinear optical processes. Although the focus is on simple waveguide structures the utilization of complex stru

  19. Scene matching based on non-linear pre-processing on reference image and sensed image

    Institute of Scientific and Technical Information of China (English)

    Zhong Sheng; Zhang Tianxu; Sang Nong

    2005-01-01

    To solve the heterogeneous image scene matching problem, a non-linear pre-processing method for the original images before intensity-based correlation is proposed. The result shows that the proper matching probability is raised greatly. Especially for the low S/N image pairs, the effect is more remarkable.

  20. Experimental observations of the characteristics of hot electron and nonlinear processes produced in special material

    Institute of Scientific and Technical Information of China (English)

    1999-01-01

    Making use of disk targets composed of several peculiar materials (foam Au, foam C8H8)and hohlraum with a special structure, experiments have been done at"Xing Guang - II" laser facility,which study the characteristics of hot electrons and therelated nonlinear processes such as StimulatedRaman Scattering (SRS), Two Plasma Decay (TPD), StimulatedBrillouin Scattering (SBS), etc.

  1. Compound Cue Processing within the Fast and Frugal Heuristics Approach in Nonlinearly Separable Environments

    Science.gov (United States)

    Garcia-Retamero, Rocio; Hoffrage, Ulrich; Dieckmann, Anja; Ramos, Manuel

    2007-01-01

    Three experiments investigated whether participants used Take The Best (TTB) Configural, a fast and frugal heuristic that processes configurations of cues when making inferences concerning which of two alternatives has a higher criterion value. Participants were presented with a compound cue that was nonlinearly separable from its elements. The…

  2. Characterizing time dependent anomalous diffusion process: A survey on fractional derivative and nonlinear models

    Science.gov (United States)

    Wei, Song; Chen, Wen; Hon, Y. C.

    2016-11-01

    This paper investigates the temporal effects in the modeling of flows through porous media and particles transport. Studies will be made among the time fractional diffusion model and two classical nonlinear diffusion models. The effects of the parameters upon the mentioned models have been studied. By simulating the sub-diffusion processes and comparing the numerical results of these models under different boundary conditions, we can conclude that the time fractional diffusion model is more suitable for simulating the sub-diffusion with steady diffusion rate; whereas the nonlinear models are more appropriate for depicting the sub-diffusion under changing diffusion rate.

  3. Analytical investigation of machining chatter by considering the nonlinearity of process damping

    Science.gov (United States)

    Ahmadi, Keivan

    2017-04-01

    In this paper, the well-established problem of self-excited vibrations in machining is revisited to include the nonlinearity of process damping at the tool and workpiece interface. Machining dynamics is modeled using a time-delayed system with nonlinear damping, and the method of averaging is used to obtain the amplitude of the resulting limit cycles. As a result, an analytical relationship is presented to establish the stability charts corresponding with arbitrary limit cycles in machining systems. The presented analytical solutions are verified using experiments and numerical solutions.

  4. 2-D nonlinear IIR-filters for image processing - An exploratory analysis

    Science.gov (United States)

    Bauer, P. H.; Sartori, M.

    1991-01-01

    A new nonlinear IIR filter structure is introduced and its deterministic properties are analyzed. It is shown to be better suited for image processing applications than its linear shift-invariant counterpart. The new structure is obtained from causality inversion of a 2D quarterplane causal linear filter with respect to the two directions of propagation. It is demonstrated, that by using this design, a nonlinear 2D lowpass filter can be constructed, which is capable of effectively suppressing Gaussian or impulse noise without destroying important image information.

  5. PREPARATION AND SECOND-ORDER OPTICAL NONLINEARITY OF NOVEL PHENOXYSILICON NETWORKS BY SOL-GEL PROCESS

    Institute of Scientific and Technical Information of China (English)

    Xiao Huang; Jian Wang; Ling-zhi Zhang; Zhi-gang Cai; Zhao-xi Lianga

    2001-01-01

    Four phenoxysilicon networks for nonlinear optical (NLO) applications were designed and prepared by an extended sol-gel process without additional H20 and catalyst. All poled polymer network films possess high second-order nonlinear optical coefficients (d33) of 10-?~10-8 esu. The investigation of NLO temporal stability at room temperature and elevated temperature (120°C) indicated that these films exhibit high d33 stability because the orientation of the chromophores are locked in the phenoxysilicon organic/inorganic networks.

  6. 2-D nonlinear IIR-filters for image processing - An exploratory analysis

    Science.gov (United States)

    Bauer, P. H.; Sartori, M.

    1991-01-01

    A new nonlinear IIR filter structure is introduced and its deterministic properties are analyzed. It is shown to be better suited for image processing applications than its linear shift-invariant counterpart. The new structure is obtained from causality inversion of a 2D quarterplane causal linear filter with respect to the two directions of propagation. It is demonstrated, that by using this design, a nonlinear 2D lowpass filter can be constructed, which is capable of effectively suppressing Gaussian or impulse noise without destroying important image information.

  7. Markov chain approach to identifying Wiener systems

    Institute of Scientific and Technical Information of China (English)

    ZHAO WenXiao; CHEN HanFu

    2012-01-01

    Identification of the Wiener system composed of an infinite impulse response (IIR) linear subsystem followed by a static nonlinearity is considered.The recursive estimates for unknown coefficients of the linear subsystem and for the values of the nonlinear function at any fixed points are given by the stochastic approximation algorithms with expanding truncations (SAAWET).With the help of properties of the Markov chain connected with the linear subsystem,all estimates derived in the paper are proved to be strongly consistent.In comparison with the existing results on the topic,the method presented in the paper simplifies the convergence analysis and requires weaker conditions.A numerical example is given,and the simulation results are consistent with the theoretical analysis.

  8. CONTROL OF NONLINEAR PROCESS USING NEURAL NETWORK BASED MODEL PREDICTIVE CONTROL

    Directory of Open Access Journals (Sweden)

    Dr.A.TRIVEDI

    2011-04-01

    Full Text Available This paper presents a Neural Network based Model Predictive Control (NNMPC strategy to control nonlinear process. Multilayer Perceptron Neural Network (MLP is chosen to represent a Nonlinear Auto Regressive with eXogenous signal (NARX model of a nonlinear system. NARX dynamic model is based on feed-forward architecture and offers good approximation capabilities along with robustness and accuracy. Based on the identified neural model, a generalized predictive control (GPC algorithm is implemented to control the composition in acontinuous stirred tank reactor (CSTR, whose parameters are optimally determined by solving quadratic performance index using well known Levenberg-Marquardt and Quasi-Newton algorithm. NNMPC is tuned by selecting few horizon parameters and weighting factor. The tracking performance of the NNMPC is tested using different amplitude function as a reference signal on CSTR application. Also the robustness and performance is tested in the presence of disturbance on random reference signal.

  9. Hydex Glass and Amorphous Silicon for Integrated Nonlinear Optical Signal Processing

    CERN Document Server

    Morandotti, Roberto

    2015-01-01

    Photonic integrated circuits that exploit nonlinear optics in order to generate and process signals all-optically have achieved performance far superior to that possible electronically - particularly with respect to speed. Although silicon-on-insulator has been the leading platform for nonlinear optics for some time, its high two-photon absorption at telecommunications wavelengths poses a fundamental limitation. We review the recent achievements based in new CMOS-compatible platforms that are better suited than SOI for nonlinear optics, focusing on amorphous silicon and Hydex glass. We highlight their potential as well as the challenges to achieving practical solutions for many key applications. These material systems have opened up many new capabilities such as on-chip optical frequency comb generation and ultrafast optical pulse generation and measurement.

  10. Nonlinear Pulse Shaping in Fibres for Pulse Generation and Optical Processing

    Directory of Open Access Journals (Sweden)

    Sonia Boscolo

    2012-01-01

    Full Text Available The development of new all-optical technologies for data processing and signal manipulation is a field of growing importance with a strong potential for numerous applications in diverse areas of modern science. Nonlinear phenomena occurring in optical fibres have many attractive features and great, but not yet fully explored, potential in signal processing. Here, we review recent progress on the use of fibre nonlinearities for the generation and shaping of optical pulses and on the applications of advanced pulse shapes in all-optical signal processing. Amongst other topics, we will discuss ultrahigh repetition rate pulse sources, the generation of parabolic shaped pulses in active and passive fibres, the generation of pulses with triangular temporal profiles, and coherent supercontinuum sources. The signal processing applications will span optical regeneration, linear distortion compensation, optical decision at the receiver in optical communication systems, spectral and temporal signal doubling, and frequency conversion.

  11. Noise can speed convergence in Markov chains.

    Science.gov (United States)

    Franzke, Brandon; Kosko, Bart

    2011-10-01

    A new theorem shows that noise can speed convergence to equilibrium in discrete finite-state Markov chains. The noise applies to the state density and helps the Markov chain explore improbable regions of the state space. The theorem ensures that a stochastic-resonance noise benefit exists for states that obey a vector-norm inequality. Such noise leads to faster convergence because the noise reduces the norm components. A corollary shows that a noise benefit still occurs if the system states obey an alternate norm inequality. This leads to a noise-benefit algorithm that requires knowledge of the steady state. An alternative blind algorithm uses only past state information to achieve a weaker noise benefit. Simulations illustrate the predicted noise benefits in three well-known Markov models. The first model is a two-parameter Ehrenfest diffusion model that shows how noise benefits can occur in the class of birth-death processes. The second model is a Wright-Fisher model of genotype drift in population genetics. The third model is a chemical reaction network of zeolite crystallization. A fourth simulation shows a convergence rate increase of 64% for states that satisfy the theorem and an increase of 53% for states that satisfy the corollary. A final simulation shows that even suboptimal noise can speed convergence if the noise applies over successive time cycles. Noise benefits tend to be sharpest in Markov models that do not converge quickly and that do not have strong absorbing states.

  12. Heterogeneous recurrence representation and quantification of dynamic transitions in continuous nonlinear processes

    Science.gov (United States)

    Chen, Yun; Yang, Hui

    2016-06-01

    Many real-world systems are evolving over time and exhibit dynamical behaviors. In order to cope with system complexity, sensing devices are commonly deployed to monitor system dynamics. Online sensing brings the proliferation of big data that are nonlinear and nonstationary. Although there is rich information on nonlinear dynamics, significant challenges remain in realizing the full potential of sensing data for system control. This paper presents a new approach of heterogeneous recurrence analysis for online monitoring and anomaly detection in nonlinear dynamic processes. A partition scheme, named as Q-tree indexing, is firstly introduced to delineate local recurrence regions in the multi-dimensional continuous state space. Further, we design a new fractal representation of state transitions among recurrence regions, and then develop new measures to quantify heterogeneous recurrence patterns. Finally, we develop a multivariate detection method for on-line monitoring and predictive control of process recurrences. Case studies show that the proposed approach not only captures heterogeneous recurrence patterns in the transformed space, but also provides effective online control charts to monitor and detect dynamical transitions in the underlying nonlinear processes.

  13. On the extendibility of partially and Markov exchangeable binary sequences

    CERN Document Server

    Di Cecco, Davide

    2009-01-01

    In [Fortini et al., Stoch. Proc. Appl. 100 (2002), 147--165] it is demonstrated that a recurrent Markov exchangeable process in the sense of Diaconis and Freedman is essentially a partially exchangeable process in the sense of de Finetti. In case of finite sequences there is not such an equivalence. We analyze both finite partially exchangeable and finite Markov exchangeable binary sequences and formulate necessary and sufficient conditions for extendibility in both cases.

  14. Complex Nonlinear Behavior in Metabolic Processes: Global Bifurcation Analysis of Escherichia coli Growth on Multiple Substrates

    Directory of Open Access Journals (Sweden)

    Hyun-Seob Song

    2013-09-01

    Full Text Available The nonlinear behavior of metabolic systems can arise from at least two different sources. One comes from the nonlinear kinetics of chemical reactions in metabolism and the other from nonlinearity associated with regulatory processes. Consequently, organisms at a constant growth rate (as experienced in a chemostat could display multiple metabolic states or display complex oscillatory behavior both with potentially serious implications to process operation. This paper explores the nonlinear behavior of a metabolic model of Escherichia coli growth on mixed substrates with sufficient detail to include regulatory features through the cybernetic postulate that metabolic regulation is the consequence of a dynamic objective function ensuring the organism’s survival. The chief source of nonlinearity arises from the optimal formulation with the metabolic state determined by a convex combination of reactions contributing to the objective function. The model for anaerobic growth of E. coli was previously examined for multiple steady states in a chemostat fed by a mixture of glucose and pyruvate substrates under very specific conditions and experimentally verified. In this article, we explore the foregoing model for nonlinear behavior over the full range of parameters, γ (the fractional concentration of glucose in the feed mixture and D (the dilution rate. The observed multiplicity is in the cybernetic variables combining elementary modes. The results show steady-state multiplicity up to seven. No Hopf bifurcation was encountered, however. Bifurcation analysis of cybernetic models is complicated by the non-differentiability of the cybernetic variables for enzyme activities. A methodology is adopted here to overcome this problem, which is applicable to more complicated metabolic networks.

  15. Stochastic seismic tomography by interacting Markov chains

    Science.gov (United States)

    Bottero, Alexis; Gesret, Alexandrine; Romary, Thomas; Noble, Mark; Maisons, Christophe

    2016-10-01

    Markov chain Monte Carlo sampling methods are widely used for non-linear Bayesian inversion where no analytical expression for the forward relation between data and model parameters is available. Contrary to the linear(ized) approaches, they naturally allow to evaluate the uncertainties on the model found. Nevertheless their use is problematic in high-dimensional model spaces especially when the computational cost of the forward problem is significant and/or the a posteriori distribution is multimodal. In this case, the chain can stay stuck in one of the modes and hence not provide an exhaustive sampling of the distribution of interest. We present here a still relatively unknown algorithm that allows interaction between several Markov chains at different temperatures. These interactions (based on importance resampling) ensure a robust sampling of any posterior distribution and thus provide a way to efficiently tackle complex fully non-linear inverse problems. The algorithm is easy to implement and is well adapted to run on parallel supercomputers. In this paper, the algorithm is first introduced and applied to a synthetic multimodal distribution in order to demonstrate its robustness and efficiency compared to a simulated annealing method. It is then applied in the framework of first arrival traveltime seismic tomography on real data recorded in the context of hydraulic fracturing. To carry out this study a wavelet-based adaptive model parametrization has been used. This allows to integrate the a priori information provided by sonic logs and to reduce optimally the dimension of the problem.

  16. Markov Decision Process for the Energy Management of Parallel Hybrid Vehicles%并联式混合动力汽车能量管理的马尔可夫决策

    Institute of Scientific and Technical Information of China (English)

    肖仁鑫; 李涛; 秦颖; 邹敢

    2012-01-01

    In order to study the energy management strategy of single-shaft parallel hybrid electric vehicles, dynamic equations of the power-train system were set up to analyze the non-aftereffect property of the required torque. To achieve the optimization objective of minimizing the oil consumption under a fixed battery capacity, Markov decision process was carried out to implement the torque allocation strategy, and the policy iteration algorithm was used to solve the Markov decision model for energy management. In addition, the Markov decision process for energy management was simulated under the condition of J1015 driving cycles and Kunming driving cycles, and executed on line. The results show that compared to the strategy based on dynamic programming, the energy management strategy based on Markov decision process, which can be implemented on line, can make the battery capacity change more smoothly, but is globally sub-optimal in oil consumption; with the new strategy, the oil consumption is increased by 1.32 L per 100 km in case of J1015 driving cycles and 1.59 L per 100 km in case of Kuming driving cycles.%为研究同轴并联式混合动力汽车的能量管理策略,建立了同轴并联式动力系统动态方程,分析了转矩需求无后效性的马尔可夫特性.在维持电池容量不变的条件下,以燃油消耗最小为优化目标,采用马尔可夫决策实施能量管理策略,并采用策略迭代方法求解了马尔可夫能量管理的转矩决策过程,在J1015工况和昆明工况进行了仿真,实现了能量管理的在线实施.结果表明,与基于动态规划的能量管理策略相比,马尔可夫决策的能量管理策略能在线实施,且电池容量变化更为平稳;在燃料消耗方面是全局次优的,在J1015行驶工况下100 km燃油消耗增加了1.32 L,在昆明行驶工况下100 km燃油消耗增加了1.59 L.

  17. Circuits and systems based on delta modulation linear, nonlinear and mixed mode processing

    CERN Document Server

    Zrilic, Djuro G

    2005-01-01

    This book is intended for students and professionals who are interested in the field of digital signal processing of delta-sigma modulated sequences. The overall focus is on the development of algorithms and circuits for linear, non-linear, and mixed mode processing of delta-sigma modulated pulse streams. The material presented here is directly relevant to applications in digital communication, DSP, instrumentation, and control.

  18. Experimental photonic generation of chirped pulses using nonlinear dispersion-based incoherent processing.

    Science.gov (United States)

    Rius, Manuel; Bolea, Mario; Mora, José; Ortega, Beatriz; Capmany, José

    2015-05-18

    We experimentally demonstrate, for the first time, a chirped microwave pulses generator based on the processing of an incoherent optical signal by means of a nonlinear dispersive element. Different capabilities have been demonstrated such as the control of the time-bandwidth product and the frequency tuning increasing the flexibility of the generated waveform compared to coherent techniques. Moreover, the use of differential detection improves considerably the limitation over the signal-to-noise ratio related to incoherent processing.

  19. Markov Karar Süreci İle Modellenen Stokastik ve Çok Amaçlı Üretim/Envanter Problemlerinin Hedef Programlama Yaklaşımı İle Çözülmesi (Solving Stochastic and Multi-Objective Production/Inventory Problems Modeled By MARKOV Decision Process with Goal Programming Approach

    Directory of Open Access Journals (Sweden)

    Aslı ÖZDEMİR

    2009-07-01

    Full Text Available To make decisions involving uncertainty while making future plans, Markov Decision Process (MDP, one of the stochastic approaches, may provide assistance to managers. Methods such as value iteration, policy iteration or linear programming can be used in the solution of MDPs when only one objective such as profit maximization or cost minimization is considered. However the decisions made by business while operating in a competition environment require considering multiple and usually conflicting objectives simultaneously. Goal programming (GP, can be used to solve such problems. The aim of this study is to provide an integrated perspective involving the utilization of MDP and GP approaches together for the solution of stochastic multi-objective decision problems. To this end the production/inventory system of a business operating in the automotive supplier industry is considered.

  20. Spectral Analysis of Markov Chains

    OpenAIRE

    2007-01-01

    The paper deals with the problem of a statistical analysis of Markov chains connected with the spectral density. We present the expressions for the function of spectral density. These expressions may be used to estimate the parameter of the Markov chain.