You should definitely know about Google’s Guest Mode

When you set Google Assistant on your Nest speaker or smart display to Guest Mode, you can rest easy that your houseguests won’t be able to access any of your sensitive information — like your calendar, search history, and other personal details.

They can still ask questions and get some information, just not your info.

To start it: Say “Hey Google, turn on guest mode.”
It’ll chime, and a little person will appear at the top right of the screen to indicate Guest Mode is on—and if you ask for personal things like the calendar, it will tell you that it can’t provide that in Guest Mode.

To turn it off: “Hey Google, exit guest mode.” 
And then you can access all your information again!