Please read the descriptions below to choose which position to apply to. In general, we expect scholars to be active stakeholders in the Research Scholars Programme, and encourage them both to set up their own structures and activities, and to adjust existing structures to suit them, in conversation with the programme managers. Will artificial agents save or destroy us? We think RSP is a good opportunity for some people who intend to go into non-research roles, where macrostrategic considerations are still of importance to this work. This leads to an exploration criterion that is both necessary and sufficient for asymptotic optimality (learning to act optimally across the entire environment class).This paper provides a formal definition of safe interruptibility and exploits the off-policy learning property to prove that either some agents are already safely interruptible, like Q-learning, or can easily be made so, like Sarsa.
Showing breadth of ideas and approaches will be an advantage, but this doesn’t have to mean choosing different research areas.No. A key aspect to this is the study of existential risks – events that endanger the survival of Earth-originating, intelligent life or that threaten to drastically and permanently destroy our potential for realising a valuable future.
The academic technical research field is very diverse, though most of the funding comes via FLI. In contrast to information gain, exploration potential takes the problem’s reward structure into account. You might also be interested in post on research scholars’ experiences of the programme, and post on where scholars are going next after leaving the Research Scholars Programme.Research scholars were co-authors on the following pieces:Assessing the Risks Posed by the Convergence of Artificial Intelligence and BiotechnologyDefence in Depth Against Human Extinction: Prevention, Response, Resilience, and Why They All MatterWho Should We Fear More: Biohackers, Disgruntled Postdocs, or Bad Governments? A Simple Risk Chain Model of BioriskThe effectiveness and perceived burden of nonpharmaceutical interventions against COVID-19 transmission: a modelling study with 41 countries (Brauner, Mindermann, Sharma, Stephenson, Gavenčiak, Johnston, Leech, Salvatier, Altman, Norman, Monrad, Besiroglu, Ge, Mikulik, Hartwick, Teh, Chindelevitch, Gal, & Beyond near and far: A new taxonomy for the AI Policy spaceRisks from Learned Optimization in Advanced Machine Learning Systems, Leike, Gal, Shevlane, & Dafoe, technical blog post, 2020)Engineered pathogens: the opportunities, risks and challengesUK Government’s approach to emerging infectious diseases and bioweapons, Bonsall, Thompson, Millett, Collyer, Lewis, Millett, Rutten, O’Brien, Rhodes, Eccleston-Turner, Bezuidenhout, Hilton, Sándor, Du Plessisa; submission to a UK Parliamentary Inquiry, 2019)Research scholars ran or co-ran several programmes, some of which attracted independent funding: for the Oxford Artificial Intelligence Society, 2019 & (separately) 2020Scholars also mentored junior researchers on a number of programmes, and taught classes, gave talks, or ran workshops at several further events. They use models of bounded and biased cognition as part of a generative model for human choices in decision problems, and infer preferences by inverting this model.Rapid developments in biotechnology and genetic engineering will pose novel risks and opportunities for humanity in the decades to come. The work that scholars do on the programme need not be macrostrategy research, or even research; the idea rather is that choices and projects are macrostrategy-driven, i.e.
The best researchers are often primarily motivated by captivation. Participants do not need to have significant research experience, but they must be able to demonstrate aptitude.Scholars spend most of their time on a combination of research, learning, applied work and meta, with priorities between these activities largely determined by the scholars themselves.Research: spending at least some time figuring out macrostrategy-driven research questions is a core part of the Research Scholars Programme.
Does the category of GCBR merit special research effort?This paper provides an overview of biotechnological extinction risk, makes some estimates for how severe the risks might be, and compares the cost-effectiveness of reducing these extinction-level risks with existing biosecurity work. Individual scholars undertake a whole range of activities, including networking and giving talks, mentoring and teaching, advising other organisations and governments, organising events and workshops, community building and project management.Meta: we think that an important part of improving one’s judgement is planning and reflection, and encourage scholars to spend time on this at many different levels, including unblocking immediate problems, project planning and evaluation, quarterly reviews of their progress overall, career planning and longer-term goals, and investing in building good long-term habits around research, productivity and wellbeing.For scholars’ own descriptions of what they spend their time on, seeWe want to provide the structure and support which will ultimately be most useful for participants.
We embed scholars in a rich intellectual environment at FHI and a wider network of long-termist researchers, enabling them to engage with new ideas and people and receive feedback on their work. Also providing access to various online resources and discussion spaces. ; our researchers are frequently invited to present their work to important audiences (e.g. This includes reading groups, discussions, salons, collaborations, workshops and other activities.
Showing breadth of ideas and approaches will be an advantage, but this doesn’t have to mean choosing different research areas.No. A key aspect to this is the study of existential risks – events that endanger the survival of Earth-originating, intelligent life or that threaten to drastically and permanently destroy our potential for realising a valuable future.
The academic technical research field is very diverse, though most of the funding comes via FLI. In contrast to information gain, exploration potential takes the problem’s reward structure into account. You might also be interested in post on research scholars’ experiences of the programme, and post on where scholars are going next after leaving the Research Scholars Programme.Research scholars were co-authors on the following pieces:Assessing the Risks Posed by the Convergence of Artificial Intelligence and BiotechnologyDefence in Depth Against Human Extinction: Prevention, Response, Resilience, and Why They All MatterWho Should We Fear More: Biohackers, Disgruntled Postdocs, or Bad Governments? A Simple Risk Chain Model of BioriskThe effectiveness and perceived burden of nonpharmaceutical interventions against COVID-19 transmission: a modelling study with 41 countries (Brauner, Mindermann, Sharma, Stephenson, Gavenčiak, Johnston, Leech, Salvatier, Altman, Norman, Monrad, Besiroglu, Ge, Mikulik, Hartwick, Teh, Chindelevitch, Gal, & Beyond near and far: A new taxonomy for the AI Policy spaceRisks from Learned Optimization in Advanced Machine Learning Systems, Leike, Gal, Shevlane, & Dafoe, technical blog post, 2020)Engineered pathogens: the opportunities, risks and challengesUK Government’s approach to emerging infectious diseases and bioweapons, Bonsall, Thompson, Millett, Collyer, Lewis, Millett, Rutten, O’Brien, Rhodes, Eccleston-Turner, Bezuidenhout, Hilton, Sándor, Du Plessisa; submission to a UK Parliamentary Inquiry, 2019)Research scholars ran or co-ran several programmes, some of which attracted independent funding: for the Oxford Artificial Intelligence Society, 2019 & (separately) 2020Scholars also mentored junior researchers on a number of programmes, and taught classes, gave talks, or ran workshops at several further events. They use models of bounded and biased cognition as part of a generative model for human choices in decision problems, and infer preferences by inverting this model.Rapid developments in biotechnology and genetic engineering will pose novel risks and opportunities for humanity in the decades to come. The work that scholars do on the programme need not be macrostrategy research, or even research; the idea rather is that choices and projects are macrostrategy-driven, i.e.
The best researchers are often primarily motivated by captivation. Participants do not need to have significant research experience, but they must be able to demonstrate aptitude.Scholars spend most of their time on a combination of research, learning, applied work and meta, with priorities between these activities largely determined by the scholars themselves.Research: spending at least some time figuring out macrostrategy-driven research questions is a core part of the Research Scholars Programme.
Does the category of GCBR merit special research effort?This paper provides an overview of biotechnological extinction risk, makes some estimates for how severe the risks might be, and compares the cost-effectiveness of reducing these extinction-level risks with existing biosecurity work. Individual scholars undertake a whole range of activities, including networking and giving talks, mentoring and teaching, advising other organisations and governments, organising events and workshops, community building and project management.Meta: we think that an important part of improving one’s judgement is planning and reflection, and encourage scholars to spend time on this at many different levels, including unblocking immediate problems, project planning and evaluation, quarterly reviews of their progress overall, career planning and longer-term goals, and investing in building good long-term habits around research, productivity and wellbeing.For scholars’ own descriptions of what they spend their time on, seeWe want to provide the structure and support which will ultimately be most useful for participants.
We embed scholars in a rich intellectual environment at FHI and a wider network of long-termist researchers, enabling them to engage with new ideas and people and receive feedback on their work. Also providing access to various online resources and discussion spaces. ; our researchers are frequently invited to present their work to important audiences (e.g. This includes reading groups, discussions, salons, collaborations, workshops and other activities.