Package: markovDP 0.99.0
markovDP: Infrastructure for Discrete-Time Markov Decision Processes (MDP)
Provides the infrastructure to work with Markov Decision Processes (MDPs) in R. The focus is on convenience in formulating MDPs, the support of sparse representations (using sparse matrices, lists and data.frames) and visualization of results. Some key components are implemented in C++ to speed up computation. Several popular solvers are implemented.
Authors:
markovDP_0.99.0.tar.gz
markovDP_0.99.0.zip(r-4.5)markovDP_0.99.0.zip(r-4.4)markovDP_0.99.0.zip(r-4.3)
markovDP_0.99.0.tgz(r-4.4-x86_64)markovDP_0.99.0.tgz(r-4.4-arm64)markovDP_0.99.0.tgz(r-4.3-x86_64)markovDP_0.99.0.tgz(r-4.3-arm64)
markovDP_0.99.0.tar.gz(r-4.5-noble)markovDP_0.99.0.tar.gz(r-4.4-noble)
markovDP_0.99.0.tgz(r-4.4-emscripten)markovDP_0.99.0.tgz(r-4.3-emscripten)
markovDP.pdf |markovDP.html✨
markovDP/json (API)
NEWS
| # Install 'markovDP' in R: |
| install.packages('markovDP', repos = c('https://mhahsler.r-universe.dev', 'https://cloud.r-project.org')) |
Bug tracker:https://github.com/mhahsler/markovdp/issues
- Cliff_walking - Cliff Walking Gridworld MDP
- DynaMaze - The Dyna Maze
- Maze - Steward Russell's 4x3 Maze Gridworld MDP
- Windy_gridworld - Windy Gridworld MDP Windy Gridworld MDP
control-theorymarkov-decision-processoptimization
Last updated 10 hours agofrom:47c32ac775. Checks:OK: 2 WARNING: 4 ERROR: 3. Indexed: yes.
| Target | Result | Date |
|---|---|---|
| Doc / Vignettes | OK | Sep 20 2024 |
| R-4.5-win-x86_64 | WARNING | Sep 20 2024 |
| R-4.5-linux-x86_64 | WARNING | Sep 20 2024 |
| R-4.4-win-x86_64 | WARNING | Sep 20 2024 |
| R-4.4-mac-x86_64 | WARNING | Sep 20 2024 |
| R-4.4-mac-aarch64 | OK | Sep 18 2024 |
| R-4.3-win-x86_64 | ERROR | Sep 20 2024 |
| R-4.3-mac-x86_64 | ERROR | Sep 20 2024 |
| R-4.3-mac-aarch64 | ERROR | Sep 20 2024 |
Exports:absorbing_statesactactionadd_policyavailable_actionsbellman_operatorcolors_continuouscolors_discretecurve_multiple_directedepoch_to_episodegreedy_actiongreedy_policygridworld_animategridworld_initgridworld_matrixgridworld_maze_MDPgridworld_pathgridworld_plotgridworld_plot_transition_graphgridworld_random_mazegridworld_rc2sgridworld_read_mazegridworld_s2rcgridworld_transition_probgridworld_transition_prob3is_solved_MDPmanual_policyMDPnormalize_MDPplot_transition_graphplot_value_functionpolicypolicy_evaluationq_valuesR_random_policyregretremove_unreachable_statesrewardreward_matrixround_stochasticsample_MDPsolve_MDPsolve_MDP_DPsolve_MDP_LPsolve_MDP_MCsolve_MDP_samplingsolve_MDP_TDstart_vectorT_transition_graphtransition_matrixunreachable_statesvalue_function
Dependencies:clicodetoolscpp11crayonforeachgluehmsigraphiteratorslatticelifecyclelpSolvemagrittrMatrixpkgconfigprettyunitsprogressR6Rcpprlangvctrs
Readme and manuals
Help Manual
| Help page | Topics |
|---|---|
| Access to Parts of the Model Description | accessors reward_matrix start_vector transition_matrix |
| Perform an Action | act |
| Choose an Action Given a Policy | action action.MDP |
| Available Actions in a State | actions available_actions |
| Add a Policy to an MDP Problem Description | add_policy |
| Cliff Walking Gridworld MDP | Cliff_walking cliff_walking |
| Default Colors for Visualization | colors colors_continuous colors_discrete |
| The Dyna Maze | DynaMaze dynamaze |
| Helper Functions for Gridworld MDPs | gridworld gridworld_animate gridworld_init gridworld_matrix gridworld_maze_MDP gridworld_path gridworld_plot gridworld_plot_transition_graph gridworld_random_maze gridworld_rc2s gridworld_read_maze gridworld_s2rc gridworld_transition_prob gridworld_transition_prob3 |
| Steward Russell's 4x3 Maze Gridworld MDP | Maze maze |
| Define an MDP Problem | epoch_to_episode is_solved_MDP MDP normalize_MDP R_ T_ |
| Extract or Create a Policy | manual_policy policy random_policy |
| Policy Evaluation | bellman_operator policy_evaluation |
| Q-Values and Greedy Policies | greedy_action greedy_policy q_values |
| Calculate the Regret of a Policy | regret |
| Calculate the Expected Reward of a Policy | reward reward.MDP |
| Round a stochastic vector or a row-stochastic matrix | round_stochastic |
| Sample Trajectories from an MDP | sample_MDP |
| Solve an MDP Problem | solve_MDP solve_MDP_DP solve_MDP_LP solve_MDP_MC solve_MDP_sampling solve_MDP_TD |
| Transition Graph | curve_multiple_directed plot_transition_graph transition_graph |
| Unreachable and Absorbing States | absorbing_states remove_unreachable_states unreachable_and_absorbing unreachable_states |
| Value Function | plot_value_function value_function |
| Windy Gridworld MDP Windy Gridworld MDP | Windy_gridworld windy_gridworld |
