You are on the right track with the song being about growing up and loosing innocence. It's basically a rite-of-passage song, especially sexually. The beanstalk represents an extremely phallic symbol that when it "grows" takes Jack to a wonderful new place full of wealth, as symbolised by the golden eggs, and culture (the harp), and also new sexual experiences - as represented by the female giant (or a female adult or fantasy). Here Freud's Oedipal complex (a son killing their father to marry their mother) comes into it - Jack killing the giantess' husband and having a relationship with the female giant.
Okay, so these are the lyrics to 'Giants in the Sky', allow me to break them down:

There are giants in the sky
There are big tall terrible giants in the sky

When you're way up high and you look below
At the world you left and the things you know
Little more than a glance is enough to show you
Just how small you are

- An obvious reference to being in a new and unfamiliar place experiencing adult things. Leaving childhood and infancy behind.

When you're way up high and you're on your own
In a world like none that you've ever known
Where the sky is lead and the earth is stone
You're free to do whatever pleases you
Exploring things you'd never dare

- Sexual references to masturbating and exploring sexual experiences.

'Cause you don't care
When suddenly there's
A big tall terrible giant at the door
A big tall lady giant sweeping the floor
And she gives you food and she gives you rest
And she draws you close to her giant breast
And you know things now that you never knew before
Not 'till the sky

- Again, more sexual references. The female giant teaching Jack about sexuality.

Only just when you've made a friend and all
And you know she's big but you don't feel small
Someone bigger than her comes along the hall
To swallow you for lunch

- Here the Oedipus Complex comes into it - the other masculine presence becoming a threat for Jack and his growing maturity, both physically and sexually. It's also a reference to be caught masturbating by your parents (eww).


And your heart is lead and your stomach stone
And you're really scared being all alone
And it's then that you miss all the things you've known
And the world you've left and the little you own
The fun is done
You steal what you can and run
And you scramble down and you look below
And the world you know begins to grow

- "The world begins to grow" this can be taken literally and metaphorically. Because of Jack's encounter with sexuality, wealth, and culture in an adult world, he know sees his own world as having so many more possibilities now.

The roof, the house, and your mother at the door
The roof, the house, and the world you never thought to explore
And you think of all of the things you've seen
And you wish that you could live in between
And you're back again only different than before
After the sky

There are giants in the sky
There are big tall terrible, awesome, scary, wonderful giants in the sky

- This last bit just reinforces that even though he has learnt a lot about himself, he is still too young to make it on his own just yet. He still desires the love and safety of his mother, especially when a larger masculine presence is after him. And even though he has returned home, his experience has changed him significantly - he has grown up and reached maturity, both mentally and sexually. The song is basically Jack telling the audience about the lessons he has learned.