Package: markovDP 0.99.0

markovDP: Infrastructure for Discrete-Time Markov Decision Processes (MDP)

Provides the infrastructure to work with Markov Decision Processes (MDPs) in R. The focus is on convenience in formulating MDPs, the support of sparse representations (using sparse matrices, lists and data.frames) and visualization of results. Some key components are implemented in C++ to speed up computation. Several popular solvers are implemented.

Authors:Michael Hahsler [aut, cph, cre]

markovDP_0.99.0.tar.gz
markovDP_0.99.0.zip(r-4.5)markovDP_0.99.0.zip(r-4.4)markovDP_0.99.0.zip(r-4.3)
markovDP_0.99.0.tgz(r-4.4-x86_64)markovDP_0.99.0.tgz(r-4.4-arm64)markovDP_0.99.0.tgz(r-4.3-x86_64)markovDP_0.99.0.tgz(r-4.3-arm64)
markovDP_0.99.0.tar.gz(r-4.5-noble)markovDP_0.99.0.tar.gz(r-4.4-noble)
markovDP_0.99.0.tgz(r-4.4-emscripten)markovDP_0.99.0.tgz(r-4.3-emscripten)
markovDP.pdf |markovDP.html
markovDP/json (API)
NEWS

# Install 'markovDP' in R:
install.packages('markovDP', repos = c('https://mhahsler.r-universe.dev', 'https://cloud.r-project.org'))

Peer review:

Bug tracker:https://github.com/mhahsler/markovdp/issues

Uses libs:
  • c++– GNU Standard C++ Library v3
Datasets:

On CRAN:

control-theorymarkov-decision-processoptimizationcpp

5.03 score 6 stars 2 scripts 78 exports 25 dependencies

Last updated 10 hours agofrom:73cf5e375c. Checks:1 OK, 8 ERROR. Indexed: yes.

TargetResultLatest binary
Doc / VignettesOKJan 18 2025
R-4.5-win-x86_64ERRORJan 18 2025
R-4.5-linux-x86_64ERRORJan 18 2025
R-4.4-win-x86_64ERRORJan 18 2025
R-4.4-mac-x86_64ERRORJan 18 2025
R-4.4-mac-aarch64ERRORJan 18 2025
R-4.3-win-x86_64ERRORJan 18 2025
R-4.3-mac-x86_64ERRORJan 18 2025
R-4.3-mac-aarch64ERRORJan 18 2025

Exports:Aabsorbing_statesactactionaction_discrepancyadd_linear_approx_Q_functionadd_policyapprox_greedy_actionapprox_greedy_policyapprox_Q_valueavailable_actionsbellman_operatorbellman_updatecolors_continuouscolors_discreteconvergence_horizoncurve_multiple_directedfind_reachable_statesgreedy_actiongreedy_policygw_animategw_initgw_matrixgw_maze_MDPgw_pathgw_plotgw_plot_transition_graphgw_random_mazegw_rc2sgw_read_mazegw_s2rcgw_transition_probgw_transition_prob_end_stategw_transition_prob_namedgw_transition_prob_sparseinduced_reward_matrixinduced_transition_matrixis_converged_MDPis_solved_MDPmanual_policyMDPnormalize_MDPP_plot_transition_graphplot_value_functionpolicypolicy_evaluationpolicy_evaluation_LPQ_randomQ_valuesQ_zeroR_random_policyregretremove_unreachable_statesrewardreward_matrixround_stochasticSsample_MDPsolve_MDPsolve_MDP_DPsolve_MDP_LPsolve_MDP_MCsolve_MDP_SAMPsolve_MDP_samplingsolve_MDP_SGDsolve_MDP_TDsolve_MDP_TDNstart_vectortransition_graphtransition_matrixunreachable_statesV_randomV_zerovalue_errorvalue_functionvisit_probability

Dependencies:clicodetoolscpp11crayonfastmapfloatforeachgluehmsigraphiteratorslatticelifecyclelpSolvemagrittrMatrixMatrixExtrapkgconfigprettyunitsprogressR6RcppRhpcBLASctlrlangvctrs

Gridworlds in Package markovDP

Rendered fromgridworlds.Rmdusingknitr::rmarkdownon Jan 18 2025.

Last update: 2025-01-17
Started: 2024-05-30

markovDP: Discrete-Time Markov Decision Processes (MDPs)

Rendered frommarkovDP.Rmdusingknitr::rmarkdownon Jan 18 2025.

Last update: 2024-11-15
Started: 2024-05-31

Readme and manuals

Help Manual

Help pageTopics
Absorbing Statesabsorbing_states
Perform an Actionact
Choose an Action Given a Policyaction action.MDP
Available Actions in a Stateavailable_actions
Bellman Update and Bellman operatorbellman_operator bellman_update
Cliff Walking Gridworld MDPCliff_walking cliff_walking
Default Colors for Visualizationcolors colors_continuous colors_discrete
Estimate the Convergence Horizon for an Infinite-Horizon MDPconvergence_horizon
The Dyna MazeDynaMaze dynamaze
Find Reachable State Space from a Transition Model Functionfind_reachable_states
Greedy Actions and Policiesgreedy_action greedy_policy
Helper Functions for Gridworld MDPsgridworld gw gw_animate gw_init gw_matrix gw_maze_MDP gw_path gw_plot gw_plot_transition_graph gw_random_maze gw_rc2s gw_read_maze gw_s2rc gw_transition_prob gw_transition_prob_end_state gw_transition_prob_named gw_transition_prob_sparse
Steward Russell's 4x3 Maze Gridworld MDPMaze maze
Define an MDP ProblemA is_converged_MDP is_solved_MDP MDP P_ R_ S
Extract, Create Add a Policy to a Modeladd_policy induced_reward_matrix induced_transition_matrix manual_policy policy random_policy
Policy Evaluationpolicy_evaluation policy_evaluation_LP
Q-ValuesQ_random Q_values Q_zero
Regret of a Policy and Related Measuresaction_discrepancy regret value_error
Calculate the Expected Reward of a Policyreward reward.MDP
Round a stochastic vector or a row-stochastic matrixround_stochastic
Sample Trajectories from an MDPsample_MDP
Solve an MDP Problemsolve_MDP
Solve MDPs using Dynamic Programmingsolve_MDP_DP
Solve MDPs using Linear Programmingsolve_MDP_LP
Solve MDPs using Monte Carlo Controlsolve_MDP_MC
Solve MDPs using Random-Samplingsolve_MDP_SAMP
Solve MDPs using Random-Samplingsolve_MDP_sampling
Episodic Semi-gradient Sarsa with Linear Function Approximationadd_linear_approx_Q_function approx_greedy_action approx_greedy_policy approx_Q_value solve_MDP_SGD
Solve MDPs using Temporal Differencingsolve_MDP_TD solve_MDP_TDN
Transition Graphcurve_multiple_directed plot_transition_graph transition_graph
Access to Parts of the Model Descriptionaccessors normalize_MDP reward_matrix start_vector transition_matrix
Unreachable Statesremove_unreachable_states unreachable_states
Value Functionplot_value_function value_function V_random V_zero
State Visit Probabilityvisit_probability
Windy Gridworld MDP Windy Gridworld MDPWindy_gridworld windy_gridworld