Package: markovDP 0.99.0
markovDP: Infrastructure for Discrete-Time Markov Decision Processes (MDP)
Provides the infrastructure to work with Markov Decision Processes (MDPs) in R. The focus is on convenience in formulating MDPs, the support of sparse representations (using sparse matrices, lists and data.frames) and visualization of results. Some key components are implemented in C++ to speed up computation. Several popular solvers are implemented.
Authors:
markovDP_0.99.0.tar.gz
markovDP_0.99.0.zip(r-4.5)markovDP_0.99.0.zip(r-4.4)markovDP_0.99.0.zip(r-4.3)
markovDP_0.99.0.tgz(r-4.4-x86_64)markovDP_0.99.0.tgz(r-4.4-arm64)markovDP_0.99.0.tgz(r-4.3-x86_64)markovDP_0.99.0.tgz(r-4.3-arm64)
markovDP_0.99.0.tar.gz(r-4.5-noble)markovDP_0.99.0.tar.gz(r-4.4-noble)
markovDP_0.99.0.tgz(r-4.4-emscripten)markovDP_0.99.0.tgz(r-4.3-emscripten)
markovDP.pdf |markovDP.html✨
markovDP/json (API)
NEWS
| # Install 'markovDP' in R: |
| install.packages('markovDP', repos = c('https://mhahsler.r-universe.dev', 'https://cloud.r-project.org')) |
Bug tracker:https://github.com/mhahsler/markovdp/issues
- Cliff_walking - Cliff Walking Gridworld MDP
- DynaMaze - The Dyna Maze
- Maze - Steward Russell's 4x3 Maze Gridworld MDP
- Windy_gridworld - Windy Gridworld MDP Windy Gridworld MDP
control-theorymarkov-decision-processoptimizationcpp
Last updated 30 days agofrom:3c52974a33. Checks:OK: 6 ERROR: 3. Indexed: yes.
| Target | Result | Date |
|---|---|---|
| Doc / Vignettes | OK | Dec 21 2024 |
| R-4.5-win-x86_64 | OK | Dec 21 2024 |
| R-4.5-linux-x86_64 | OK | Dec 21 2024 |
| R-4.4-win-x86_64 | OK | Dec 21 2024 |
| R-4.4-mac-x86_64 | OK | Dec 21 2024 |
| R-4.4-mac-aarch64 | OK | Dec 21 2024 |
| R-4.3-win-x86_64 | ERROR | Dec 21 2024 |
| R-4.3-mac-x86_64 | ERROR | Dec 21 2024 |
| R-4.3-mac-aarch64 | ERROR | Dec 21 2024 |
Exports:Aabsorbing_statesactactionaction_discrepancyadd_policyavailable_actionsbellman_operatorbellman_updatecolors_continuouscolors_discretecurve_multiple_directedepoch_to_episodefind_reachable_statesgreedy_actiongreedy_policygw_animategw_initgw_matrixgw_maze_MDPgw_pathgw_plotgw_plot_transition_graphgw_random_mazegw_rc2sgw_read_mazegw_s2rcgw_transition_probgw_transition_prob_end_stateinduced_reward_matrixinduced_transition_matrixis_converged_MDPis_solved_MDPmanual_policyMDPnormalize_MDPP_plot_transition_graphplot_value_functionpolicypolicy_evaluationpolicy_evaluation_LPQ_randomQ_valuesQ_zeroR_random_policyregretremove_unreachable_statesrewardreward_matrixround_stochasticSsample_MDPsolve_MDPsolve_MDP_DPsolve_MDP_LPsolve_MDP_MCsolve_MDP_samplingsolve_MDP_TDstart_vectortransition_graphtransition_matrixunreachable_statesV_randomV_zerovalue_errorvalue_functionvisit_probability
Dependencies:clicodetoolscpp11crayonfastmapforeachgluehmsigraphiteratorslatticelifecyclelpSolvemagrittrMatrixpkgconfigprettyunitsprogressR6Rcpprlangvctrs
Readme and manuals
Help Manual
| Help page | Topics |
|---|---|
| Absorbing States | absorbing_states |
| Access to Parts of the Model Description | accessors normalize_MDP reward_matrix start_vector transition_matrix |
| Perform an Action | act |
| Choose an Action Given a Policy | action action.MDP |
| Available Actions in a State | available_actions |
| Bellman Update and Bellman operator | bellman_operator bellman_update |
| Cliff Walking Gridworld MDP | Cliff_walking cliff_walking |
| Default Colors for Visualization | colors colors_continuous colors_discrete |
| The Dyna Maze | DynaMaze dynamaze |
| Find Reachable State Space from a Transition Model Function | find_reachable_states |
| Greedy Actions and Policies | greedy_action greedy_policy |
| Helper Functions for Gridworld MDPs | gridworld gw gw_animate gw_init gw_matrix gw_maze_MDP gw_path gw_plot gw_plot_transition_graph gw_random_maze gw_rc2s gw_read_maze gw_s2rc gw_transition_prob gw_transition_prob_end_state |
| Steward Russell's 4x3 Maze Gridworld MDP | Maze maze |
| Define an MDP Problem | A epoch_to_episode is_converged_MDP is_solved_MDP MDP P_ R_ S |
| Extract, Create Add a Policy to a Model | add_policy induced_reward_matrix induced_transition_matrix manual_policy policy random_policy |
| Policy Evaluation | policy_evaluation policy_evaluation_LP |
| Q-Values | Q_random Q_values Q_zero |
| Regret of a Policy and Related Measures | action_discrepancy regret value_error |
| Calculate the Expected Reward of a Policy | reward reward.MDP |
| Round a stochastic vector or a row-stochastic matrix | round_stochastic |
| Sample Trajectories from an MDP | sample_MDP |
| Solve an MDP Problem | solve_MDP solve_MDP_DP solve_MDP_LP solve_MDP_MC solve_MDP_sampling solve_MDP_TD |
| Transition Graph | curve_multiple_directed plot_transition_graph transition_graph |
| Unreachable States | remove_unreachable_states unreachable_states |
| Value Function | plot_value_function value_function V_random V_zero |
| State Visit Probability | visit_probability |
| Windy Gridworld MDP Windy Gridworld MDP | Windy_gridworld windy_gridworld |
