Package: markovDP 0.99.0

markovDP: Infrastructure for Discrete-Time Markov Decision Processes (MDP)

Provides the infrastructure to work with Markov Decision Processes (MDPs) in R. The focus is on convenience in formulating MDPs, the support of sparse representations (using sparse matrices, lists and data.frames) and visualization of results. Some key components are implemented in C++ to speed up computation. Several popular solvers are implemented.

Authors:Michael Hahsler [aut, cph, cre]

markovDP_0.99.0.tar.gz
markovDP_0.99.0.zip(r-4.5)markovDP_0.99.0.zip(r-4.4)markovDP_0.99.0.zip(r-4.3)
markovDP_0.99.0.tgz(r-4.4-x86_64)markovDP_0.99.0.tgz(r-4.4-arm64)markovDP_0.99.0.tgz(r-4.3-x86_64)markovDP_0.99.0.tgz(r-4.3-arm64)
markovDP_0.99.0.tar.gz(r-4.5-noble)markovDP_0.99.0.tar.gz(r-4.4-noble)
markovDP_0.99.0.tgz(r-4.4-emscripten)markovDP_0.99.0.tgz(r-4.3-emscripten)
markovDP.pdf |markovDP.html
markovDP/json (API)
NEWS

# Install 'markovDP' in R:
install.packages('markovDP', repos = c('https://mhahsler.r-universe.dev', 'https://cloud.r-project.org'))

Peer review:

Bug tracker:https://github.com/mhahsler/markovdp/issues

Uses libs:
  • c++– GNU Standard C++ Library v3
Datasets:

On CRAN:

control-theorymarkov-decision-processoptimization

54 exports 5 stars 1.24 score 21 dependencies 1 scripts

Last updated 10 hours agofrom:47c32ac775. Checks:OK: 2 WARNING: 4 ERROR: 3. Indexed: yes.

TargetResultDate
Doc / VignettesOKSep 20 2024
R-4.5-win-x86_64WARNINGSep 20 2024
R-4.5-linux-x86_64WARNINGSep 20 2024
R-4.4-win-x86_64WARNINGSep 20 2024
R-4.4-mac-x86_64WARNINGSep 20 2024
R-4.4-mac-aarch64OKSep 18 2024
R-4.3-win-x86_64ERRORSep 20 2024
R-4.3-mac-x86_64ERRORSep 20 2024
R-4.3-mac-aarch64ERRORSep 20 2024

Exports:absorbing_statesactactionadd_policyavailable_actionsbellman_operatorcolors_continuouscolors_discretecurve_multiple_directedepoch_to_episodegreedy_actiongreedy_policygridworld_animategridworld_initgridworld_matrixgridworld_maze_MDPgridworld_pathgridworld_plotgridworld_plot_transition_graphgridworld_random_mazegridworld_rc2sgridworld_read_mazegridworld_s2rcgridworld_transition_probgridworld_transition_prob3is_solved_MDPmanual_policyMDPnormalize_MDPplot_transition_graphplot_value_functionpolicypolicy_evaluationq_valuesR_random_policyregretremove_unreachable_statesrewardreward_matrixround_stochasticsample_MDPsolve_MDPsolve_MDP_DPsolve_MDP_LPsolve_MDP_MCsolve_MDP_samplingsolve_MDP_TDstart_vectorT_transition_graphtransition_matrixunreachable_statesvalue_function

Dependencies:clicodetoolscpp11crayonforeachgluehmsigraphiteratorslatticelifecyclelpSolvemagrittrMatrixpkgconfigprettyunitsprogressR6Rcpprlangvctrs

Gridworlds in Package markovDP

Rendered fromgridworlds.Rmdusingknitr::rmarkdownon Sep 20 2024.

Last update: 2024-09-17
Started: 2024-05-30

markovDP: Discrete-Time Markov Decision Processes (MDPs)

Rendered frommarkovDP.Rmdusingknitr::rmarkdownon Sep 20 2024.

Last update: 2024-09-10
Started: 2024-05-31

Readme and manuals

Help Manual

Help pageTopics
Access to Parts of the Model Descriptionaccessors reward_matrix start_vector transition_matrix
Perform an Actionact
Choose an Action Given a Policyaction action.MDP
Available Actions in a Stateactions available_actions
Add a Policy to an MDP Problem Descriptionadd_policy
Cliff Walking Gridworld MDPCliff_walking cliff_walking
Default Colors for Visualizationcolors colors_continuous colors_discrete
The Dyna MazeDynaMaze dynamaze
Helper Functions for Gridworld MDPsgridworld gridworld_animate gridworld_init gridworld_matrix gridworld_maze_MDP gridworld_path gridworld_plot gridworld_plot_transition_graph gridworld_random_maze gridworld_rc2s gridworld_read_maze gridworld_s2rc gridworld_transition_prob gridworld_transition_prob3
Steward Russell's 4x3 Maze Gridworld MDPMaze maze
Define an MDP Problemepoch_to_episode is_solved_MDP MDP normalize_MDP R_ T_
Extract or Create a Policymanual_policy policy random_policy
Policy Evaluationbellman_operator policy_evaluation
Q-Values and Greedy Policiesgreedy_action greedy_policy q_values
Calculate the Regret of a Policyregret
Calculate the Expected Reward of a Policyreward reward.MDP
Round a stochastic vector or a row-stochastic matrixround_stochastic
Sample Trajectories from an MDPsample_MDP
Solve an MDP Problemsolve_MDP solve_MDP_DP solve_MDP_LP solve_MDP_MC solve_MDP_sampling solve_MDP_TD
Transition Graphcurve_multiple_directed plot_transition_graph transition_graph
Unreachable and Absorbing Statesabsorbing_states remove_unreachable_states unreachable_and_absorbing unreachable_states
Value Functionplot_value_function value_function
Windy Gridworld MDP Windy Gridworld MDPWindy_gridworld windy_gridworld